[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:33:37 dist-test-slave-cq7c krb5kdc[6999](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:33:37 dist-test-slave-cq7c krb5kdc[6999](info): set up 2 sockets
Sep 24 11:33:37 dist-test-slave-cq7c krb5kdc[6999](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 11:33:38.237200 6992 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.002s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Sep 24 11:33:38 dist-test-slave-cq7c krb5kdc[6999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:33:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:33:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.212.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
I20260924 11:33:38.429093 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:39193
--webserver_interface=127.6.212.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:33873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:39193
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:33:38.769477 7012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:33:38.769822 7012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:33:38.769997 7012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:33:38.777415 7012 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:33:38.777572 7012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:33:38.777679 7012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:33:38.777783 7012 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:33:38.777868 7012 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:33:38.788417 7012 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:33873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:39193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:39193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.212.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:33:38.789803 7012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:33:38.791167 7012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:33:38.804080 7017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:33:38.804272 7020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:33:38.805073 7018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:33:38.805826 7012 server_base.cc:1034] running on GCE node
I20260924 11:33:38.806903 7012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:33:38.808573 7012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:33:38.809816 7012 hybrid_clock.cc:648] HybridClock initialized: now 1790249618809772 us; error 63 us; skew 500 ppm
I20260924 11:33:38.810194 7012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:33:38 dist-test-slave-cq7c krb5kdc[6999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249618, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:33:38.813503 7012 init.cc:377] Logged in from keytab as kudu/127.6.212.62@KRBTEST.COM (short username kudu)
I20260924 11:33:38.815083 7012 webserver.cc:460] Webserver started at http://127.6.212.62:41793/ using document root <none> and password file <none>
I20260924 11:33:38.815876 7012 fs_manager.cc:359] Metadata directory not provided
I20260924 11:33:38.816030 7012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:33:38.816337 7012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:33:38.819566 7012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "1bad744a8c5e4234b3847824a9f684a8"
format_stamp: "Formatted at 2026-09-24 11:33:38 on dist-test-slave-cq7c"
server_key: "5a1b2061f4183a26bbf627444d379565"
server_key_iv: "8689bbbb55d66c12fe474ad5d705c452"
server_key_version: "encryptionkey@0"
I20260924 11:33:38.820461 7012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "1bad744a8c5e4234b3847824a9f684a8"
format_stamp: "Formatted at 2026-09-24 11:33:38 on dist-test-slave-cq7c"
server_key: "5a1b2061f4183a26bbf627444d379565"
server_key_iv: "8689bbbb55d66c12fe474ad5d705c452"
server_key_version: "encryptionkey@0"
I20260924 11:33:38.825855 7012 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 11:33:38.829717 7027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:33:38.831058 7012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 11:33:38.831256 7012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "1bad744a8c5e4234b3847824a9f684a8"
format_stamp: "Formatted at 2026-09-24 11:33:38 on dist-test-slave-cq7c"
server_key: "5a1b2061f4183a26bbf627444d379565"
server_key_iv: "8689bbbb55d66c12fe474ad5d705c452"
server_key_version: "encryptionkey@0"
I20260924 11:33:38.831601 7012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:33:38.856254 7012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:33:38.861284 7012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:33:38.861622 7012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:33:38.882999 7012 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.62:39193
I20260924 11:33:38.883100 7078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.62:39193 every 8 connection(s)
I20260924 11:33:38.884467 7012 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:33:38.886785 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 7012
I20260924 11:33:38.887092 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:33:38.888353 7079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:33:38.889166 6992 external_mini_cluster.cc:1428] Setting key 70310a4bde32100c91dc0d6e671dbf4f
I20260924 11:33:38.899446 7079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bad744a8c5e4234b3847824a9f684a8: Bootstrap starting.
I20260924 11:33:38.903227 7079 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bad744a8c5e4234b3847824a9f684a8: Neither blocks nor log segments found. Creating new log.
I20260924 11:33:38.904717 7079 log.cc:824] T 00000000000000000000000000000000 P 1bad744a8c5e4234b3847824a9f684a8: Log is configured to *not* fsync() on all Append() calls
I20260924 11:33:38.907696 7079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bad744a8c5e4234b3847824a9f684a8: No bootstrap required, opened a new log
I20260924 11:33:38.917273 7079 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bad744a8c5e4234b3847824a9f684a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bad744a8c5e4234b3847824a9f684a8" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 39193 } }
I20260924 11:33:38.917660 7079 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bad744a8c5e4234b3847824a9f684a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:33:38.917788 7079 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bad744a8c5e4234b3847824a9f684a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bad744a8c5e4234b3847824a9f684a8, State: Initialized, Role: FOLLOWER
I20260924 11:33:38.918429 7079 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bad744a8c5e4234b3847824a9f684a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bad744a8c5e4234b3847824a9f684a8" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 39193 } }
I20260924 11:33:38.918645 7079 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bad744a8c5e4234b3847824a9f684a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:33:38.918799 7079 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bad744a8c5e4234b3847824a9f684a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:33:38.918977 7079 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bad744a8c5e4234b3847824a9f684a8 [term 0 FOLLOWER]: Advancing to term 1
Sep 24 11:33:38 dist-test-slave-cq7c krb5kdc[6999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:33:38.921964 7079 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bad744a8c5e4234b3847824a9f684a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bad744a8c5e4234b3847824a9f684a8" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 39193 } }
I20260924 11:33:38.922640 7079 leader_election.cc:302] T 00000000000000000000000000000000 P 1bad744a8c5e4234b3847824a9f684a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bad744a8c5e4234b3847824a9f684a8; no voters:
I20260924 11:33:38.923470 7079 leader_election.cc:288] T 00000000000000000000000000000000 P 1bad744a8c5e4234b3847824a9f684a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:33:38.923697 7084 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bad744a8c5e4234b3847824a9f684a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:33:38.924364 7084 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bad744a8c5e4234b3847824a9f684a8 [term 1 LEADER]: Becoming Leader. State: Replica: 1bad744a8c5e4234b3847824a9f684a8, State: Running, Role: LEADER
I20260924 11:33:38.924975 7084 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bad744a8c5e4234b3847824a9f684a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bad744a8c5e4234b3847824a9f684a8" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 39193 } }
I20260924 11:33:38.927743 7086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bad744a8c5e4234b3847824a9f684a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1bad744a8c5e4234b3847824a9f684a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bad744a8c5e4234b3847824a9f684a8" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 39193 } } }
I20260924 11:33:38.928151 7086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bad744a8c5e4234b3847824a9f684a8 [sys.catalog]: This master's current role is: LEADER
I20260924 11:33:38.930220 7085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bad744a8c5e4234b3847824a9f684a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bad744a8c5e4234b3847824a9f684a8. Latest consensus state: current_term: 1 leader_uuid: "1bad744a8c5e4234b3847824a9f684a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bad744a8c5e4234b3847824a9f684a8" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 39193 } } }
I20260924 11:33:38.930495 7085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bad744a8c5e4234b3847824a9f684a8 [sys.catalog]: This master's current role is: LEADER
I20260924 11:33:38.932857 7079 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bad744a8c5e4234b3847824a9f684a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:33:38.935523 7081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:38.892882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33176 (local address 127.6.212.62:39193)
0924 11:33:38.893994 (+ 1112us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:38.894013 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:38.895316 (+ 1303us) server_negotiation.cc:407] Connection header received
0924 11:33:38.904719 (+ 9403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:38.904748 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:38.905317 (+ 569us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:38.905700 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:33:38.908910 (+ 3210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:38.911271 (+ 2361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:38.913087 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:38.914233 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:38.914518 (+ 285us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:38.921274 (+ 6756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:33:38.921283 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:33:38.921299 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:33:38.921341 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:33:38.926626 (+ 5285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:38.928094 (+ 1468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:38.928106 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:38.928118 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:38.928398 (+ 280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:38.929101 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:38.929109 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:38.929114 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:38.929984 (+ 870us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:33:38.933110 (+ 3126us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:38.933374 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 11:33:38.933805 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":838,"thread_start_us":587,"threads_started":1}
I20260924 11:33:38.939033 7088 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:33:38.944900 7088 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:33:38.955580 7088 catalog_manager.cc:1347] Generated new cluster ID: 325825396c6b4c6c83ee95ba2b5669fa
I20260924 11:33:38.955706 7088 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:33:38.970991 7088 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:33:38.972811 7088 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:33:38.984083 7088 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bad744a8c5e4234b3847824a9f684a8: Generated new TSK 0
I20260924 11:33:38.985157 7088 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.212.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab.
Entry for principal kudu/127.6.212.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab.
I20260924 11:33:39.039312 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.1:0
--local_ip_for_outbound_sockets=127.6.212.1
--webserver_interface=127.6.212.1
--webserver_port=0
--tserver_master_addrs=127.6.212.62:39193
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab
--principal=kudu/127.6.212.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:33873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:33:39.389317 7106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:33:39.389597 7106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:33:39.389752 7106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:33:39.397115 7106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:33:39.397230 7106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:33:39.397346 7106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.1
I20260924 11:33:39.407933 7106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:33873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab
--principal=kudu/127.6.212.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.212.1
--webserver_port=0
--tserver_master_addrs=127.6.212.62:39193
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.1
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:33:39.409160 7106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:33:39.410524 7106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:33:39.423947 7111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:33:39.424108 7112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:33:39.424158 7114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:33:39.425334 7106 server_base.cc:1034] running on GCE node
I20260924 11:33:39.425966 7106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:33:39.426964 7106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:33:39.428188 7106 hybrid_clock.cc:648] HybridClock initialized: now 1790249619428136 us; error 47 us; skew 500 ppm
I20260924 11:33:39.428524 7106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:33:39 dist-test-slave-cq7c krb5kdc[6999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249619, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:33:39.431772 7106 init.cc:377] Logged in from keytab as kudu/127.6.212.1@KRBTEST.COM (short username kudu)
I20260924 11:33:39.433604 7106 webserver.cc:460] Webserver started at http://127.6.212.1:46281/ using document root <none> and password file <none>
I20260924 11:33:39.434379 7106 fs_manager.cc:359] Metadata directory not provided
I20260924 11:33:39.434513 7106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:33:39.434887 7106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:33:39.437922 7106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-0/data/instance:
uuid: "6c25859bd5424e03b9195ac8953edc82"
format_stamp: "Formatted at 2026-09-24 11:33:39 on dist-test-slave-cq7c"
server_key: "34dd1d9dc0131625dc1bcaf06fa307aa"
server_key_iv: "359ef7090ef3cec3fc54e7de76378519"
server_key_version: "encryptionkey@0"
I20260924 11:33:39.438681 7106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance:
uuid: "6c25859bd5424e03b9195ac8953edc82"
format_stamp: "Formatted at 2026-09-24 11:33:39 on dist-test-slave-cq7c"
server_key: "34dd1d9dc0131625dc1bcaf06fa307aa"
server_key_iv: "359ef7090ef3cec3fc54e7de76378519"
server_key_version: "encryptionkey@0"
I20260924 11:33:39.444077 7106 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 11:33:39.447628 7121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:33:39.448915 7106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:33:39.449102 7106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-0/wal
uuid: "6c25859bd5424e03b9195ac8953edc82"
format_stamp: "Formatted at 2026-09-24 11:33:39 on dist-test-slave-cq7c"
server_key: "34dd1d9dc0131625dc1bcaf06fa307aa"
server_key_iv: "359ef7090ef3cec3fc54e7de76378519"
server_key_version: "encryptionkey@0"
I20260924 11:33:39.449302 7106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:33:39.469564 7106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:33:39.473719 7106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:33:39.474092 7106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:33:39.475107 7106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:33:39.476491 7106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:33:39.476605 7106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:33:39.476822 7106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:33:39.476902 7106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:33:39.514405 7106 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.1:33997
I20260924 11:33:39.514452 7233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.1:33997 every 8 connection(s)
I20260924 11:33:39.515739 7106 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
I20260924 11:33:39.517270 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 7106
I20260924 11:33:39.517505 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance
I20260924 11:33:39.518127 6992 external_mini_cluster.cc:1428] Setting key 1ef737b7ea393c0ff631e0da45892d80
Sep 24 11:33:39 dist-test-slave-cq7c krb5kdc[6999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249619, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.212.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.2@KRBTEST.COM" created.
I20260924 11:33:39.539499 7236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:39.518371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:42557 (local address 127.6.212.62:39193)
0924 11:33:39.518849 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:39.518857 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:39.520419 (+ 1562us) server_negotiation.cc:407] Connection header received
0924 11:33:39.522258 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:39.522263 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:39.522361 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:39.522495 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:33:39.524839 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:39.525588 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:39.528210 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:39.528605 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:39.528705 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:39.532983 (+ 4278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:33:39.532991 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:33:39.532996 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:33:39.533041 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:33:39.536135 (+ 3094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:39.536834 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:39.536838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:39.536840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:39.536998 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:39.537417 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:39.537424 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:39.537427 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:39.537858 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:33:39.537998 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:39.538965 (+ 967us) server_negotiation.cc:299] Negotiation successful
0924 11:33:39.539140 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":236,"threads_started":1}
I20260924 11:33:39.541014 7237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:39.518952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:39193 (local address 127.6.212.1:42557)
0924 11:33:39.520272 (+ 1320us) negotiation.cc:107] Waiting for socket to connect
0924 11:33:39.520315 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 11:33:39.521981 (+ 1666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:33:39.522730 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:33:39.522739 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:33:39.523827 (+ 1088us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:33:39.524655 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:39.524682 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:39.525748 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:39.525757 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:39.528071 (+ 2314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:39.528088 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:39.528844 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:39.528853 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:39.529201 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:39.529919 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:33:39.529968 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:33:39.532711 (+ 2743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:33:39.536256 (+ 3545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:33:39.536263 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:33:39.536276 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:33:39.536711 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:33:39.537092 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:33:39.537095 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:33:39.537098 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:33:39.537274 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:33:39.537996 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:33:39.538003 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:33:39.538483 (+ 480us) client_negotiation.cc:769] Sending connection context
0924 11:33:39.538808 (+ 325us) client_negotiation.cc:240] Negotiation successful
0924 11:33:39.539474 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1042,"thread_start_us":793,"threads_started":1}
I20260924 11:33:39.542709 7234 heartbeater.cc:347] Connected to a master server at 127.6.212.62:39193
I20260924 11:33:39.543072 7234 heartbeater.cc:464] Registering TS with master...
I20260924 11:33:39.543915 7234 heartbeater.cc:511] Master 127.6.212.62:39193 requested a full tablet report, sending...
I20260924 11:33:39.546412 7044 ts_manager.cc:194] Registered new tserver with Master: 6c25859bd5424e03b9195ac8953edc82 (127.6.212.1:33997)
I20260924 11:33:39.548692 7044 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.1@KRBTEST.COM'} at 127.6.212.1:42557
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab.
Entry for principal kudu/127.6.212.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab.
I20260924 11:33:39.563002 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.2:0
--local_ip_for_outbound_sockets=127.6.212.2
--webserver_interface=127.6.212.2
--webserver_port=0
--tserver_master_addrs=127.6.212.62:39193
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab
--principal=kudu/127.6.212.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:33873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:33:39.909092 7240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:33:39.909360 7240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:33:39.909603 7240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:33:39.916893 7240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:33:39.917019 7240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:33:39.917210 7240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.2
I20260924 11:33:39.927484 7240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:33873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab
--principal=kudu/127.6.212.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.212.2
--webserver_port=0
--tserver_master_addrs=127.6.212.62:39193
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.2
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:33:39.928797 7240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:33:39.930166 7240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:33:39.944553 7245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:33:39.944577 7248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:33:39.944514 7246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:33:39.945263 7240 server_base.cc:1034] running on GCE node
I20260924 11:33:39.945990 7240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:33:39.947134 7240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:33:39.948432 7240 hybrid_clock.cc:648] HybridClock initialized: now 1790249619948317 us; error 48 us; skew 500 ppm
I20260924 11:33:39.948808 7240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:33:39 dist-test-slave-cq7c krb5kdc[6999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249619, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:33:39.952239 7240 init.cc:377] Logged in from keytab as kudu/127.6.212.2@KRBTEST.COM (short username kudu)
I20260924 11:33:39.953852 7240 webserver.cc:460] Webserver started at http://127.6.212.2:35061/ using document root <none> and password file <none>
I20260924 11:33:39.954591 7240 fs_manager.cc:359] Metadata directory not provided
I20260924 11:33:39.954743 7240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:33:39.955067 7240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:33:39.958568 7240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-1/data/instance:
uuid: "33732fbbfd7a47419bb3c305f183c2de"
format_stamp: "Formatted at 2026-09-24 11:33:39 on dist-test-slave-cq7c"
server_key: "d8ba0c9a97f8546d25b921bf4b9f4bbc"
server_key_iv: "855a2be826a5d574e9a53d1ef78e4733"
server_key_version: "encryptionkey@0"
I20260924 11:33:39.959570 7240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance:
uuid: "33732fbbfd7a47419bb3c305f183c2de"
format_stamp: "Formatted at 2026-09-24 11:33:39 on dist-test-slave-cq7c"
server_key: "d8ba0c9a97f8546d25b921bf4b9f4bbc"
server_key_iv: "855a2be826a5d574e9a53d1ef78e4733"
server_key_version: "encryptionkey@0"
I20260924 11:33:39.965121 7240 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 11:33:39.968868 7255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:33:39.970263 7240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:33:39.970516 7240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-1/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-1/wal
uuid: "33732fbbfd7a47419bb3c305f183c2de"
format_stamp: "Formatted at 2026-09-24 11:33:39 on dist-test-slave-cq7c"
server_key: "d8ba0c9a97f8546d25b921bf4b9f4bbc"
server_key_iv: "855a2be826a5d574e9a53d1ef78e4733"
server_key_version: "encryptionkey@0"
I20260924 11:33:39.971135 7240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:33:39.994609 7240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:33:39.998831 7240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:33:39.999141 7240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:33:40.000202 7240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:33:40.001803 7240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:33:40.001924 7240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:33:40.002134 7240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:33:40.002249 7240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:33:40.040474 7240 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.2:42691
I20260924 11:33:40.040602 7367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.2:42691 every 8 connection(s)
I20260924 11:33:40.041764 7240 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
I20260924 11:33:40.049952 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 7240
I20260924 11:33:40.050161 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance
I20260924 11:33:40.050679 6992 external_mini_cluster.cc:1428] Setting key f29026b0bdd27e470f930b9561b56196
Sep 24 11:33:40 dist-test-slave-cq7c krb5kdc[6999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249619, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.2@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:33:40.064956 7370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:40.044939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:38451 (local address 127.6.212.62:39193)
0924 11:33:40.045395 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:40.045400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:40.046499 (+ 1099us) server_negotiation.cc:407] Connection header received
0924 11:33:40.048577 (+ 2078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:40.048582 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:40.048686 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:40.048831 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:33:40.050464 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:40.051348 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:40.053612 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:40.054091 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:40.054235 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:40.058198 (+ 3963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:33:40.058204 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:33:40.058211 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:33:40.058254 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:33:40.061550 (+ 3296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:40.062573 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:40.062576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:40.062578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:40.062699 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:40.063191 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:40.063193 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:40.063195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:40.063506 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:33:40.063618 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:40.064549 (+ 931us) server_negotiation.cc:299] Negotiation successful
0924 11:33:40.064703 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":221,"threads_started":1}
I20260924 11:33:40.066577 7371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:40.045562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:39193 (local address 127.6.212.2:38451)
0924 11:33:40.046325 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0924 11:33:40.046370 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 11:33:40.048305 (+ 1935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:33:40.049095 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:33:40.049104 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:33:40.049612 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:33:40.050294 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:40.050313 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:40.051557 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:40.051562 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:40.053475 (+ 1913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:40.053487 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:40.054318 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:40.054326 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:40.054592 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:40.055314 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:33:40.055337 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:33:40.057950 (+ 2613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:33:40.061692 (+ 3742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:33:40.061704 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:33:40.061725 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:33:40.062455 (+ 730us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:33:40.062812 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:33:40.062820 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:33:40.062828 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:33:40.063080 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:33:40.063640 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:33:40.063652 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:33:40.064152 (+ 500us) client_negotiation.cc:769] Sending connection context
0924 11:33:40.064500 (+ 348us) client_negotiation.cc:240] Negotiation successful
0924 11:33:40.065142 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":325,"threads_started":1}
I20260924 11:33:40.068140 7368 heartbeater.cc:347] Connected to a master server at 127.6.212.62:39193
I20260924 11:33:40.068485 7368 heartbeater.cc:464] Registering TS with master...
I20260924 11:33:40.069386 7368 heartbeater.cc:511] Master 127.6.212.62:39193 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.212.3@KRBTEST.COM; defaulting to no policy
I20260924 11:33:40.071075 7044 ts_manager.cc:194] Registered new tserver with Master: 33732fbbfd7a47419bb3c305f183c2de (127.6.212.2:42691)
I20260924 11:33:40.072153 7044 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.2@KRBTEST.COM'} at 127.6.212.2:38451
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab.
Entry for principal kudu/127.6.212.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab.
I20260924 11:33:40.094211 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.3:0
--local_ip_for_outbound_sockets=127.6.212.3
--webserver_interface=127.6.212.3
--webserver_port=0
--tserver_master_addrs=127.6.212.62:39193
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab
--principal=kudu/127.6.212.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:33873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:33:40.442687 7374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:33:40.443109 7374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:33:40.443287 7374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:33:40.450613 7374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:33:40.450757 7374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:33:40.450945 7374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.3
I20260924 11:33:40.460984 7374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:33873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab
--principal=kudu/127.6.212.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.212.3
--webserver_port=0
--tserver_master_addrs=127.6.212.62:39193
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.3
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:33:40.462396 7374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:33:40.464003 7374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:33:40.478107 7379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:33:40.478080 7382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:33:40.478323 7380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:33:40.479053 7374 server_base.cc:1034] running on GCE node
I20260924 11:33:40.479768 7374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:33:40.480733 7374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:33:40.482045 7374 hybrid_clock.cc:648] HybridClock initialized: now 1790249620481993 us; error 44 us; skew 500 ppm
I20260924 11:33:40.482401 7374 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:33:40 dist-test-slave-cq7c krb5kdc[6999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249620, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:33:40.485723 7374 init.cc:377] Logged in from keytab as kudu/127.6.212.3@KRBTEST.COM (short username kudu)
I20260924 11:33:40.487236 7374 webserver.cc:460] Webserver started at http://127.6.212.3:38911/ using document root <none> and password file <none>
I20260924 11:33:40.487947 7374 fs_manager.cc:359] Metadata directory not provided
I20260924 11:33:40.488094 7374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:33:40.488400 7374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:33:40.491451 7374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-2/data/instance:
uuid: "a7610eec02ec4d888e6907763fa96c91"
format_stamp: "Formatted at 2026-09-24 11:33:40 on dist-test-slave-cq7c"
server_key: "64f77095c29d89534075e184c434161c"
server_key_iv: "48c5d78cf7941dd2f69d8500807fadb6"
server_key_version: "encryptionkey@0"
I20260924 11:33:40.492480 7374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance:
uuid: "a7610eec02ec4d888e6907763fa96c91"
format_stamp: "Formatted at 2026-09-24 11:33:40 on dist-test-slave-cq7c"
server_key: "64f77095c29d89534075e184c434161c"
server_key_iv: "48c5d78cf7941dd2f69d8500807fadb6"
server_key_version: "encryptionkey@0"
I20260924 11:33:40.497730 7374 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260924 11:33:40.501909 7389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:33:40.503540 7374 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:33:40.503717 7374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-2/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-2/wal
uuid: "a7610eec02ec4d888e6907763fa96c91"
format_stamp: "Formatted at 2026-09-24 11:33:40 on dist-test-slave-cq7c"
server_key: "64f77095c29d89534075e184c434161c"
server_key_iv: "48c5d78cf7941dd2f69d8500807fadb6"
server_key_version: "encryptionkey@0"
I20260924 11:33:40.503921 7374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:33:40.521257 7374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:33:40.525238 7374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:33:40.525642 7374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:33:40.527004 7374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:33:40.528347 7374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:33:40.528456 7374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:33:40.528618 7374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:33:40.528692 7374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:33:40.551990 7234 heartbeater.cc:503] Master 127.6.212.62:39193 was elected leader, sending a full tablet report...
I20260924 11:33:40.567757 7374 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.3:41089
I20260924 11:33:40.567785 7501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.3:41089 every 8 connection(s)
I20260924 11:33:40.569016 7374 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
I20260924 11:33:40.571543 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 7374
I20260924 11:33:40.571761 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance
I20260924 11:33:40.572340 6992 external_mini_cluster.cc:1428] Setting key 4edd5abfe8b7a3796a5fcbaeee1e3c36
Sep 24 11:33:40 dist-test-slave-cq7c krb5kdc[6999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249620, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.3@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:33:40.589986 7504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:40.571869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:34873 (local address 127.6.212.62:39193)
0924 11:33:40.572320 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:40.572326 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:40.573472 (+ 1146us) server_negotiation.cc:407] Connection header received
0924 11:33:40.575485 (+ 2013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:40.575490 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:40.575598 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:40.575722 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:33:40.577237 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:40.578021 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:40.579594 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:40.579977 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:40.580110 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:40.584169 (+ 4059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:33:40.584174 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:33:40.584177 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:33:40.584214 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:33:40.586703 (+ 2489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:40.587738 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:40.587742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:40.587744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:40.587855 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:40.588299 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:40.588303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:40.588312 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:40.588567 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:33:40.588726 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:40.589391 (+ 665us) server_negotiation.cc:299] Negotiation successful
0924 11:33:40.589606 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":222,"threads_started":1}
I20260924 11:33:40.590993 7505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:40.572504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:39193 (local address 127.6.212.3:34873)
0924 11:33:40.573306 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0924 11:33:40.573350 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 11:33:40.575134 (+ 1784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:33:40.575956 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:33:40.575965 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:33:40.576525 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:33:40.577072 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:40.577090 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:40.578182 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:40.578188 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:40.579456 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:40.579468 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:40.580203 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:40.580209 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:40.580443 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:40.581211 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:33:40.581232 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:33:40.583971 (+ 2739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:33:40.586846 (+ 2875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:33:40.586857 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:33:40.586879 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:33:40.587608 (+ 729us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:33:40.587973 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:33:40.587980 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:33:40.587983 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:33:40.588177 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:33:40.588692 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:33:40.588700 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:33:40.589045 (+ 345us) client_negotiation.cc:769] Sending connection context
0924 11:33:40.589316 (+ 271us) client_negotiation.cc:240] Negotiation successful
0924 11:33:40.589817 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":329,"threads_started":1}
I20260924 11:33:40.592288 7502 heartbeater.cc:347] Connected to a master server at 127.6.212.62:39193
I20260924 11:33:40.592760 7502 heartbeater.cc:464] Registering TS with master...
I20260924 11:33:40.593544 7502 heartbeater.cc:511] Master 127.6.212.62:39193 requested a full tablet report, sending...
I20260924 11:33:40.594923 7044 ts_manager.cc:194] Registered new tserver with Master: a7610eec02ec4d888e6907763fa96c91 (127.6.212.3:41089)
I20260924 11:33:40.595757 7044 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.3@KRBTEST.COM'} at 127.6.212.3:34873
I20260924 11:33:40.600762 6992 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 11:33:40 dist-test-slave-cq7c krb5kdc[6999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249620, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 11:33:40 dist-test-slave-cq7c krb5kdc[6999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249620, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.212.1@KRBTEST.COM
I20260924 11:33:40.645819 7510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:40.629856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56258 (local address 127.6.212.1:33997)
0924 11:33:40.630545 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:40.630556 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:40.630588 (+ 32us) server_negotiation.cc:407] Connection header received
0924 11:33:40.630800 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:40.630808 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:40.631036 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:40.631202 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:33:40.633178 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:40.634296 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:40.635667 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:40.636270 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:40.636504 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:40.640199 (+ 3695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:33:40.640209 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:33:40.640223 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:33:40.640257 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:33:40.642914 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:40.643700 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:40.643706 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:40.643709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:40.643826 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:40.644235 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:40.644242 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:40.644244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:40.644581 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:33:40.644878 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:40.645160 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 11:33:40.645411 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":346,"threads_started":1}
W20260924 11:33:40.647094 7149 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56258
Sep 24 11:33:40 dist-test-slave-cq7c krb5kdc[6999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 11:33:40 dist-test-slave-cq7c krb5kdc[6999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.1@KRBTEST.COM
I20260924 11:33:40.689749 7510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:40.676291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56268 (local address 127.6.212.1:33997)
0924 11:33:40.676544 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:40.676551 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:40.676823 (+ 272us) server_negotiation.cc:407] Connection header received
0924 11:33:40.677053 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:40.677062 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:40.677153 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:40.677264 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:33:40.678547 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:40.679209 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:40.680558 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:40.680964 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:40.681104 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:40.684741 (+ 3637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:33:40.684746 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:33:40.684749 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:33:40.684778 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:33:40.687275 (+ 2497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:40.687902 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:40.687909 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:40.687912 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:40.688034 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:40.688397 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:40.688401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:40.688404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:40.688646 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:33:40.688877 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:40.689219 (+ 342us) server_negotiation.cc:299] Negotiation successful
0924 11:33:40.689428 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 11:33:40.691465 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 7106
I20260924 11:33:40.704985 7510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:40.692369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56276 (local address 127.6.212.1:33997)
0924 11:33:40.692527 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:40.692531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:40.692911 (+ 380us) server_negotiation.cc:407] Connection header received
0924 11:33:40.693039 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:40.693042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:40.693091 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:40.693164 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:33:40.694402 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:40.695069 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:40.696499 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:40.696910 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:40.697002 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:40.699490 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:33:40.699495 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:33:40.699498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:33:40.699525 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:33:40.701913 (+ 2388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:40.702616 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:40.702619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:40.702622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:40.702750 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:40.703603 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:40.703606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:40.703608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:40.703842 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:33:40.704012 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:40.704414 (+ 402us) server_negotiation.cc:299] Negotiation successful
0924 11:33:40.704623 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 11:33:40.705724 7229 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:33:40.789788 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 7106
I20260924 11:33:40.806536 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 7240
Sep 24 11:33:40 dist-test-slave-cq7c krb5kdc[6999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.2@KRBTEST.COM
I20260924 11:33:40.821686 7517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:40.807547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38342 (local address 127.6.212.2:42691)
0924 11:33:40.808097 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:40.808106 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:40.808139 (+ 33us) server_negotiation.cc:407] Connection header received
0924 11:33:40.808204 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:40.808211 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:40.808413 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:40.808606 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:33:40.809664 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:40.810610 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:40.811849 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:40.812367 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:40.812469 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:40.816277 (+ 3808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:33:40.816286 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:33:40.816300 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:33:40.816329 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:33:40.819212 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:40.819871 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:40.819877 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:40.819879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:40.820006 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:40.820349 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:40.820352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:40.820354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:40.820706 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:33:40.820910 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:40.821190 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 11:33:40.821370 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":266,"threads_started":1}
I20260924 11:33:40.822914 7363 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:33:40.905021 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 7240
I20260924 11:33:40.927145 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 7374
Sep 24 11:33:40 dist-test-slave-cq7c krb5kdc[6999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.3@KRBTEST.COM
I20260924 11:33:40.945305 7519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:40.928129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40894 (local address 127.6.212.3:41089)
0924 11:33:40.928723 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:40.928731 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:40.928757 (+ 26us) server_negotiation.cc:407] Connection header received
0924 11:33:40.928817 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:40.928824 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:40.929036 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:40.929214 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:33:40.930896 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:40.932362 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:40.933888 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:40.934491 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:40.934635 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:40.938771 (+ 4136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:33:40.938781 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:33:40.938795 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:33:40.938826 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:33:40.941766 (+ 2940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:40.942699 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:40.942708 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:40.942711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:40.942851 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:40.943272 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:40.943280 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:40.943283 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:40.943884 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:33:40.944263 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:40.944589 (+ 326us) server_negotiation.cc:299] Negotiation successful
0924 11:33:40.944907 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":312,"threads_started":1}
I20260924 11:33:40.946650 7497 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:33:41.030009 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 7374
I20260924 11:33:41.046729 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 7012
I20260924 11:33:41.048015 7074 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:33:41.151506 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 7012
2026-09-24T11:33:41Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4052 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:33:41 dist-test-slave-cq7c krb5kdc[7524](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:33:41 dist-test-slave-cq7c krb5kdc[7524](info): set up 2 sockets
Sep 24 11:33:41 dist-test-slave-cq7c krb5kdc[7524](info): commencing operation
krb5kdc: starting...
W20260924 11:33:42.243289 6992 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Sep 24 11:33:42 dist-test-slave-cq7c krb5kdc[7524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:33:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:33:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.212.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
I20260924 11:33:42.423480 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:40673
--webserver_interface=127.6.212.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:39779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:40673
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:33:42.745784 7537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:33:42.746071 7537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:33:42.746249 7537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:33:42.753721 7537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:33:42.753824 7537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:33:42.753916 7537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:33:42.754006 7537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:33:42.754063 7537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:33:42.763767 7537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:39779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:40673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:40673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.212.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:33:42.765107 7537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:33:42.766438 7537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:33:42.778060 7542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:33:42.778132 7545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:33:42.778098 7543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:33:42.778995 7537 server_base.cc:1034] running on GCE node
I20260924 11:33:42.779881 7537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:33:42.781430 7537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:33:42.782706 7537 hybrid_clock.cc:648] HybridClock initialized: now 1790249622782556 us; error 142 us; skew 500 ppm
I20260924 11:33:42.783027 7537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:33:42 dist-test-slave-cq7c krb5kdc[7524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249622, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:33:42.791122 7537 init.cc:377] Logged in from keytab as kudu/127.6.212.62@KRBTEST.COM (short username kudu)
I20260924 11:33:42.792846 7537 webserver.cc:460] Webserver started at http://127.6.212.62:34341/ using document root <none> and password file <none>
I20260924 11:33:42.793581 7537 fs_manager.cc:359] Metadata directory not provided
I20260924 11:33:42.793726 7537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:33:42.794035 7537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:33:42.797024 7537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "45ec79ddf6a64927845f0a66eb6767ad"
format_stamp: "Formatted at 2026-09-24 11:33:42 on dist-test-slave-cq7c"
server_key: "0caeb8c254610b9c8be6d34e9a3945fd"
server_key_iv: "d2fbed8a671510d2b6a8a788a4042be1"
server_key_version: "encryptionkey@0"
I20260924 11:33:42.797771 7537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "45ec79ddf6a64927845f0a66eb6767ad"
format_stamp: "Formatted at 2026-09-24 11:33:42 on dist-test-slave-cq7c"
server_key: "0caeb8c254610b9c8be6d34e9a3945fd"
server_key_iv: "d2fbed8a671510d2b6a8a788a4042be1"
server_key_version: "encryptionkey@0"
I20260924 11:33:42.803056 7537 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260924 11:33:42.806589 7552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:33:42.807997 7537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:33:42.808179 7537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "45ec79ddf6a64927845f0a66eb6767ad"
format_stamp: "Formatted at 2026-09-24 11:33:42 on dist-test-slave-cq7c"
server_key: "0caeb8c254610b9c8be6d34e9a3945fd"
server_key_iv: "d2fbed8a671510d2b6a8a788a4042be1"
server_key_version: "encryptionkey@0"
I20260924 11:33:42.808375 7537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:33:42.829358 7537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:33:42.833587 7537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:33:42.833865 7537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:33:42.852224 7537 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.62:40673
I20260924 11:33:42.852254 7603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.62:40673 every 8 connection(s)
I20260924 11:33:42.853585 7537 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:33:42.857300 7604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:33:42.859997 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 7537
I20260924 11:33:42.860221 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:33:42.861001 6992 external_mini_cluster.cc:1428] Setting key 268492e87e4b21b6a1ccf964b0136fd7
I20260924 11:33:42.869167 7604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45ec79ddf6a64927845f0a66eb6767ad: Bootstrap starting.
I20260924 11:33:42.872270 7604 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45ec79ddf6a64927845f0a66eb6767ad: Neither blocks nor log segments found. Creating new log.
I20260924 11:33:42.873246 7604 log.cc:824] T 00000000000000000000000000000000 P 45ec79ddf6a64927845f0a66eb6767ad: Log is configured to *not* fsync() on all Append() calls
Sep 24 11:33:42 dist-test-slave-cq7c krb5kdc[7524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:33:42.876304 7604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45ec79ddf6a64927845f0a66eb6767ad: No bootstrap required, opened a new log
I20260924 11:33:42.884557 7607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:42.863231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34572 (local address 127.6.212.62:40673)
0924 11:33:42.864025 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:42.864037 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:42.864077 (+ 40us) server_negotiation.cc:407] Connection header received
0924 11:33:42.865671 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:42.865694 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:42.866139 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:42.866540 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:33:42.868075 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:42.869408 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:42.870621 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:42.871278 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:42.871528 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:42.875660 (+ 4132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:33:42.875674 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:33:42.875695 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:33:42.875742 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:33:42.879864 (+ 4122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:42.880662 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:42.880670 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:42.880679 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:42.880889 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:42.881233 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:42.881237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:42.881241 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:42.881937 (+ 696us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:33:42.882267 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:42.882669 (+ 402us) server_negotiation.cc:299] Negotiation successful
0924 11:33:42.883140 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":329,"threads_started":1}
I20260924 11:33:42.886878 7604 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45ec79ddf6a64927845f0a66eb6767ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45ec79ddf6a64927845f0a66eb6767ad" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 40673 } }
I20260924 11:33:42.887290 7604 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45ec79ddf6a64927845f0a66eb6767ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:33:42.887445 7604 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45ec79ddf6a64927845f0a66eb6767ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45ec79ddf6a64927845f0a66eb6767ad, State: Initialized, Role: FOLLOWER
I20260924 11:33:42.888063 7604 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45ec79ddf6a64927845f0a66eb6767ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45ec79ddf6a64927845f0a66eb6767ad" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 40673 } }
I20260924 11:33:42.888309 7604 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45ec79ddf6a64927845f0a66eb6767ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:33:42.888454 7604 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45ec79ddf6a64927845f0a66eb6767ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:33:42.888641 7604 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45ec79ddf6a64927845f0a66eb6767ad [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:33:42.890815 7604 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45ec79ddf6a64927845f0a66eb6767ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45ec79ddf6a64927845f0a66eb6767ad" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 40673 } }
I20260924 11:33:42.891368 7604 leader_election.cc:302] T 00000000000000000000000000000000 P 45ec79ddf6a64927845f0a66eb6767ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45ec79ddf6a64927845f0a66eb6767ad; no voters:
I20260924 11:33:42.892074 7604 leader_election.cc:288] T 00000000000000000000000000000000 P 45ec79ddf6a64927845f0a66eb6767ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:33:42.892179 7609 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45ec79ddf6a64927845f0a66eb6767ad [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:33:42.892947 7609 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45ec79ddf6a64927845f0a66eb6767ad [term 1 LEADER]: Becoming Leader. State: Replica: 45ec79ddf6a64927845f0a66eb6767ad, State: Running, Role: LEADER
I20260924 11:33:42.893476 7609 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45ec79ddf6a64927845f0a66eb6767ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45ec79ddf6a64927845f0a66eb6767ad" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 40673 } }
I20260924 11:33:42.894237 7604 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45ec79ddf6a64927845f0a66eb6767ad [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:33:42.896489 7611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45ec79ddf6a64927845f0a66eb6767ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45ec79ddf6a64927845f0a66eb6767ad. Latest consensus state: current_term: 1 leader_uuid: "45ec79ddf6a64927845f0a66eb6767ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45ec79ddf6a64927845f0a66eb6767ad" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 40673 } } }
I20260924 11:33:42.896937 7611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45ec79ddf6a64927845f0a66eb6767ad [sys.catalog]: This master's current role is: LEADER
I20260924 11:33:42.898123 7617 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:33:42.898120 7610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45ec79ddf6a64927845f0a66eb6767ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45ec79ddf6a64927845f0a66eb6767ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45ec79ddf6a64927845f0a66eb6767ad" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 40673 } } }
I20260924 11:33:42.899557 7610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45ec79ddf6a64927845f0a66eb6767ad [sys.catalog]: This master's current role is: LEADER
I20260924 11:33:42.903852 7617 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:33:42.912977 7617 catalog_manager.cc:1347] Generated new cluster ID: 4260a1beab3546ccb256cb0c55cd1cb7
I20260924 11:33:42.913096 7617 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:33:42.945343 7617 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:33:42.947245 7617 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:33:42.963882 7617 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45ec79ddf6a64927845f0a66eb6767ad: Generated new TSK 0
I20260924 11:33:42.965016 7617 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.212.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab.
Entry for principal kudu/127.6.212.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab.
I20260924 11:33:43.021011 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.1:0
--local_ip_for_outbound_sockets=127.6.212.1
--webserver_interface=127.6.212.1
--webserver_port=0
--tserver_master_addrs=127.6.212.62:40673
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab
--principal=kudu/127.6.212.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:39779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:33:43.341320 7630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:33:43.341660 7630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:33:43.341825 7630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:33:43.348994 7630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:33:43.349139 7630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:33:43.349349 7630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.1
I20260924 11:33:43.358778 7630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:39779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab
--principal=kudu/127.6.212.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.212.1
--webserver_port=0
--tserver_master_addrs=127.6.212.62:40673
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.1
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:33:43.359972 7630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:33:43.361213 7630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:33:43.374883 7638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:33:43.374857 7636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:33:43.375211 7635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:33:43.380863 7630 server_base.cc:1034] running on GCE node
I20260924 11:33:43.381502 7630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:33:43.382472 7630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:33:43.383709 7630 hybrid_clock.cc:648] HybridClock initialized: now 1790249623383664 us; error 44 us; skew 500 ppm
I20260924 11:33:43.384034 7630 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:33:43 dist-test-slave-cq7c krb5kdc[7524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249623, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:33:43.387132 7630 init.cc:377] Logged in from keytab as kudu/127.6.212.1@KRBTEST.COM (short username kudu)
I20260924 11:33:43.388702 7630 webserver.cc:460] Webserver started at http://127.6.212.1:36613/ using document root <none> and password file <none>
I20260924 11:33:43.389420 7630 fs_manager.cc:359] Metadata directory not provided
I20260924 11:33:43.389572 7630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:33:43.389879 7630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:33:43.392789 7630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-0/data/instance:
uuid: "5a81749df1664cd8999b5cabe1cfd247"
format_stamp: "Formatted at 2026-09-24 11:33:43 on dist-test-slave-cq7c"
server_key: "91762c683e946d1104a09361551ea2ba"
server_key_iv: "eaca24ddf1d36001db4c417b6ba9fa50"
server_key_version: "encryptionkey@0"
I20260924 11:33:43.393554 7630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance:
uuid: "5a81749df1664cd8999b5cabe1cfd247"
format_stamp: "Formatted at 2026-09-24 11:33:43 on dist-test-slave-cq7c"
server_key: "91762c683e946d1104a09361551ea2ba"
server_key_iv: "eaca24ddf1d36001db4c417b6ba9fa50"
server_key_version: "encryptionkey@0"
I20260924 11:33:43.398700 7630 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260924 11:33:43.402284 7645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:33:43.403669 7630 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:33:43.403833 7630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-0/wal
uuid: "5a81749df1664cd8999b5cabe1cfd247"
format_stamp: "Formatted at 2026-09-24 11:33:43 on dist-test-slave-cq7c"
server_key: "91762c683e946d1104a09361551ea2ba"
server_key_iv: "eaca24ddf1d36001db4c417b6ba9fa50"
server_key_version: "encryptionkey@0"
I20260924 11:33:43.404042 7630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:33:43.433725 7630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:33:43.437752 7630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:33:43.438047 7630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:33:43.439003 7630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:33:43.440395 7630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:33:43.440512 7630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:33:43.440660 7630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:33:43.440739 7630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:33:43.473816 7630 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.1:36413
I20260924 11:33:43.473867 7757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.1:36413 every 8 connection(s)
I20260924 11:33:43.475179 7630 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
I20260924 11:33:43.478907 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 7630
I20260924 11:33:43.479131 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance
I20260924 11:33:43.479818 6992 external_mini_cluster.cc:1428] Setting key bb5c064214be473b2e8ab94b7f348890
Sep 24 11:33:43 dist-test-slave-cq7c krb5kdc[7524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249623, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.212.2@KRBTEST.COM; defaulting to no policy
I20260924 11:33:43.497480 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:43.477806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:36569 (local address 127.6.212.62:40673)
0924 11:33:43.478210 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:43.478215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:43.479397 (+ 1182us) server_negotiation.cc:407] Connection header received
0924 11:33:43.481727 (+ 2330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:43.481730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:43.481789 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:43.481934 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:33:43.483402 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:43.484559 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:43.486141 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:43.486826 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:43.486974 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:43.490435 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:33:43.490444 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:33:43.490452 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:33:43.490495 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:33:43.494194 (+ 3699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:43.494898 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:43.494904 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:43.494909 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:43.495058 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:43.495524 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:43.495531 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:43.495535 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:43.495927 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:33:43.496077 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:43.496813 (+ 736us) server_negotiation.cc:299] Negotiation successful
0924 11:33:43.497030 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":201,"threads_started":1}
I20260924 11:33:43.498232 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:43.478404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:40673 (local address 127.6.212.1:36569)
0924 11:33:43.479199 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0924 11:33:43.479252 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 11:33:43.481398 (+ 2146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:33:43.482142 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:33:43.482150 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:33:43.482652 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:33:43.483201 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:43.483218 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:43.484722 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:43.484726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:43.486005 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:43.486016 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:43.487024 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:43.487028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:43.487219 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:43.487957 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:33:43.487979 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:33:43.490208 (+ 2229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:33:43.494356 (+ 4148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:33:43.494363 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:33:43.494376 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:33:43.494784 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:33:43.495181 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:33:43.495184 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:33:43.495186 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:33:43.495405 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:33:43.496055 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:33:43.496061 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:33:43.496401 (+ 340us) client_negotiation.cc:769] Sending connection context
0924 11:33:43.496637 (+ 236us) client_negotiation.cc:240] Negotiation successful
0924 11:33:43.497017 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":353,"threads_started":1}
I20260924 11:33:43.499948 7758 heartbeater.cc:347] Connected to a master server at 127.6.212.62:40673
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.2@KRBTEST.COM" created.
I20260924 11:33:43.500243 7758 heartbeater.cc:464] Registering TS with master...
I20260924 11:33:43.500942 7758 heartbeater.cc:511] Master 127.6.212.62:40673 requested a full tablet report, sending...
I20260924 11:33:43.503254 7569 ts_manager.cc:194] Registered new tserver with Master: 5a81749df1664cd8999b5cabe1cfd247 (127.6.212.1:36413)
I20260924 11:33:43.504941 7569 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.1@KRBTEST.COM'} at 127.6.212.1:36569
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab.
Entry for principal kudu/127.6.212.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab.
I20260924 11:33:43.524307 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.2:0
--local_ip_for_outbound_sockets=127.6.212.2
--webserver_interface=127.6.212.2
--webserver_port=0
--tserver_master_addrs=127.6.212.62:40673
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab
--principal=kudu/127.6.212.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:39779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:33:43.847028 7764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:33:43.847458 7764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:33:43.847630 7764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:33:43.857342 7764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:33:43.857484 7764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:33:43.857659 7764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.2
I20260924 11:33:43.869505 7764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:39779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab
--principal=kudu/127.6.212.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.212.2
--webserver_port=0
--tserver_master_addrs=127.6.212.62:40673
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.2
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:33:43.870833 7764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:33:43.872303 7764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:33:43.885458 7770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:33:43.885488 7769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:33:43.885653 7772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:33:43.886250 7764 server_base.cc:1034] running on GCE node
I20260924 11:33:43.886833 7764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:33:43.887827 7764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:33:43.889027 7764 hybrid_clock.cc:648] HybridClock initialized: now 1790249623888972 us; error 43 us; skew 500 ppm
I20260924 11:33:43.889341 7764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:33:43 dist-test-slave-cq7c krb5kdc[7524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249623, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:33:43.892505 7764 init.cc:377] Logged in from keytab as kudu/127.6.212.2@KRBTEST.COM (short username kudu)
I20260924 11:33:43.893958 7764 webserver.cc:460] Webserver started at http://127.6.212.2:34307/ using document root <none> and password file <none>
I20260924 11:33:43.894721 7764 fs_manager.cc:359] Metadata directory not provided
I20260924 11:33:43.894861 7764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:33:43.895201 7764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:33:43.898092 7764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-1/data/instance:
uuid: "c33ef45fa8e94450841709e411f7ece8"
format_stamp: "Formatted at 2026-09-24 11:33:43 on dist-test-slave-cq7c"
server_key: "0e86479f769cb245d0c0c7a1c9806e9d"
server_key_iv: "010556641443f675d1f608894e04244f"
server_key_version: "encryptionkey@0"
I20260924 11:33:43.899012 7764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance:
uuid: "c33ef45fa8e94450841709e411f7ece8"
format_stamp: "Formatted at 2026-09-24 11:33:43 on dist-test-slave-cq7c"
server_key: "0e86479f769cb245d0c0c7a1c9806e9d"
server_key_iv: "010556641443f675d1f608894e04244f"
server_key_version: "encryptionkey@0"
I20260924 11:33:43.904179 7764 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.002s
I20260924 11:33:43.907685 7779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:33:43.909024 7764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 11:33:43.909191 7764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-1/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-1/wal
uuid: "c33ef45fa8e94450841709e411f7ece8"
format_stamp: "Formatted at 2026-09-24 11:33:43 on dist-test-slave-cq7c"
server_key: "0e86479f769cb245d0c0c7a1c9806e9d"
server_key_iv: "010556641443f675d1f608894e04244f"
server_key_version: "encryptionkey@0"
I20260924 11:33:43.909377 7764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:33:43.926153 7764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:33:43.930339 7764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:33:43.930672 7764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:33:43.931751 7764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:33:43.933254 7764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:33:43.933369 7764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:33:43.933537 7764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:33:43.933617 7764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:33:43.969215 7764 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.2:41079
I20260924 11:33:43.969314 7891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.2:41079 every 8 connection(s)
I20260924 11:33:43.970642 7764 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
I20260924 11:33:43.980444 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 7764
I20260924 11:33:43.980634 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance
I20260924 11:33:43.981067 6992 external_mini_cluster.cc:1428] Setting key 24ac6db55cb6986ffaeaed8be3aa44b7
Sep 24 11:33:43 dist-test-slave-cq7c krb5kdc[7524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249623, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.2@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:33:43.993060 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:43.973595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:49155 (local address 127.6.212.62:40673)
0924 11:33:43.973799 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:43.973821 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:43.975090 (+ 1269us) server_negotiation.cc:407] Connection header received
0924 11:33:43.977093 (+ 2003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:43.977100 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:43.977194 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:43.977317 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:33:43.978921 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:43.979724 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:43.981563 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:43.982034 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:43.982195 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:43.986475 (+ 4280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:33:43.986485 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:33:43.986494 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:33:43.986544 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:33:43.989629 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:43.990580 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:43.990585 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:43.990587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:43.990715 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:43.991187 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:43.991189 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:43.991191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:43.991611 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:33:43.991721 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:43.992653 (+ 932us) server_negotiation.cc:299] Negotiation successful
0924 11:33:43.992818 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 11:33:43.994778 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:43.974005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:40673 (local address 127.6.212.2:49155)
0924 11:33:43.974913 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0924 11:33:43.974957 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 11:33:43.976757 (+ 1800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:33:43.977575 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:33:43.977584 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:33:43.978104 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:33:43.978736 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:43.978757 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:43.979907 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:43.979914 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:43.981397 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:43.981410 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:43.982303 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:43.982321 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:43.982565 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:43.983263 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:33:43.983296 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:33:43.986230 (+ 2934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:33:43.989774 (+ 3544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:33:43.989785 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:33:43.989805 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:33:43.990435 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:33:43.990829 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:33:43.990837 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:33:43.990844 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:33:43.991079 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:33:43.991740 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:33:43.991751 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:33:43.992245 (+ 494us) client_negotiation.cc:769] Sending connection context
0924 11:33:43.992573 (+ 328us) client_negotiation.cc:240] Negotiation successful
0924 11:33:43.993160 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":414,"threads_started":1}
I20260924 11:33:43.996438 7892 heartbeater.cc:347] Connected to a master server at 127.6.212.62:40673
I20260924 11:33:43.996789 7892 heartbeater.cc:464] Registering TS with master...
I20260924 11:33:43.997673 7892 heartbeater.cc:511] Master 127.6.212.62:40673 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.212.3@KRBTEST.COM; defaulting to no policy
I20260924 11:33:43.999475 7569 ts_manager.cc:194] Registered new tserver with Master: c33ef45fa8e94450841709e411f7ece8 (127.6.212.2:41079)
I20260924 11:33:44.000366 7569 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.2@KRBTEST.COM'} at 127.6.212.2:49155
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab.
Entry for principal kudu/127.6.212.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab.
I20260924 11:33:44.031785 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.3:0
--local_ip_for_outbound_sockets=127.6.212.3
--webserver_interface=127.6.212.3
--webserver_port=0
--tserver_master_addrs=127.6.212.62:40673
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab
--principal=kudu/127.6.212.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:39779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:33:44.352656 7897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:33:44.352977 7897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:33:44.353142 7897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:33:44.360411 7897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:33:44.360554 7897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:33:44.360733 7897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.3
I20260924 11:33:44.370296 7897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:39779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab
--principal=kudu/127.6.212.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.212.3
--webserver_port=0
--tserver_master_addrs=127.6.212.62:40673
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.3
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:33:44.371629 7897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:33:44.372922 7897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:33:44.385977 7902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:33:44.385951 7905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:33:44.385926 7903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:33:44.386822 7897 server_base.cc:1034] running on GCE node
I20260924 11:33:44.387425 7897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:33:44.388327 7897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:33:44.389525 7897 hybrid_clock.cc:648] HybridClock initialized: now 1790249624389491 us; error 31 us; skew 500 ppm
I20260924 11:33:44.389792 7897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:33:44 dist-test-slave-cq7c krb5kdc[7524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249624, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:33:44.392891 7897 init.cc:377] Logged in from keytab as kudu/127.6.212.3@KRBTEST.COM (short username kudu)
I20260924 11:33:44.394317 7897 webserver.cc:460] Webserver started at http://127.6.212.3:43427/ using document root <none> and password file <none>
I20260924 11:33:44.395004 7897 fs_manager.cc:359] Metadata directory not provided
I20260924 11:33:44.395128 7897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:33:44.395500 7897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:33:44.398342 7897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-2/data/instance:
uuid: "2450c43689164a16b30fb7d6f95d5473"
format_stamp: "Formatted at 2026-09-24 11:33:44 on dist-test-slave-cq7c"
server_key: "d1e7f4fed1b45944995af7d3306ca718"
server_key_iv: "44e50bba8e9bf0e887b5613fff0c6a2a"
server_key_version: "encryptionkey@0"
I20260924 11:33:44.399210 7897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance:
uuid: "2450c43689164a16b30fb7d6f95d5473"
format_stamp: "Formatted at 2026-09-24 11:33:44 on dist-test-slave-cq7c"
server_key: "d1e7f4fed1b45944995af7d3306ca718"
server_key_iv: "44e50bba8e9bf0e887b5613fff0c6a2a"
server_key_version: "encryptionkey@0"
I20260924 11:33:44.404438 7897 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260924 11:33:44.407974 7912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:33:44.409464 7897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 11:33:44.409648 7897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-2/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-2/wal
uuid: "2450c43689164a16b30fb7d6f95d5473"
format_stamp: "Formatted at 2026-09-24 11:33:44 on dist-test-slave-cq7c"
server_key: "d1e7f4fed1b45944995af7d3306ca718"
server_key_iv: "44e50bba8e9bf0e887b5613fff0c6a2a"
server_key_version: "encryptionkey@0"
I20260924 11:33:44.409849 7897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:33:44.422369 7897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:33:44.426419 7897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:33:44.426724 7897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:33:44.427743 7897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:33:44.429091 7897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:33:44.429168 7897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:33:44.429406 7897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:33:44.429461 7897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:33:44.460345 7897 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.3:33453
I20260924 11:33:44.460384 8024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.3:33453 every 8 connection(s)
I20260924 11:33:44.461712 7897 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
I20260924 11:33:44.469388 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 7897
I20260924 11:33:44.469607 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance
I20260924 11:33:44.470168 6992 external_mini_cluster.cc:1428] Setting key fbcdded4fb9e736eb370ddf91a468d32
Sep 24 11:33:44 dist-test-slave-cq7c krb5kdc[7524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249624, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.3@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:33:44.485173 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:44.464373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:41497 (local address 127.6.212.62:40673)
0924 11:33:44.464596 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:44.464601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:44.465788 (+ 1187us) server_negotiation.cc:407] Connection header received
0924 11:33:44.467946 (+ 2158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:44.467952 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:44.468061 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:44.468183 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:33:44.471030 (+ 2847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:44.472142 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:44.474587 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:44.474974 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:44.475078 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:44.479194 (+ 4116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:33:44.479199 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:33:44.479202 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:33:44.479235 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:33:44.481945 (+ 2710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:44.482716 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:44.482721 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:44.482725 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:44.482857 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:44.483272 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:44.483279 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:44.483284 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:44.483670 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:33:44.483832 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:44.484600 (+ 768us) server_negotiation.cc:299] Negotiation successful
0924 11:33:44.484812 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260924 11:33:44.486326 8027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:44.464789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:40673 (local address 127.6.212.3:41497)
0924 11:33:44.465580 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0924 11:33:44.465630 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 11:33:44.467649 (+ 2019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:33:44.468529 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:33:44.468542 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:33:44.469854 (+ 1312us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:33:44.470829 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:44.470852 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:44.472419 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:44.472423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:44.474427 (+ 2004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:44.474452 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:44.475203 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:44.475210 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:44.475561 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:44.476296 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:33:44.476327 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:33:44.478971 (+ 2644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:33:44.482141 (+ 3170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:33:44.482150 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:33:44.482166 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:33:44.482586 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:33:44.482966 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:33:44.482972 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:33:44.482975 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:33:44.483161 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:33:44.483803 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:33:44.483811 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:33:44.484200 (+ 389us) client_negotiation.cc:769] Sending connection context
0924 11:33:44.484499 (+ 299us) client_negotiation.cc:240] Negotiation successful
0924 11:33:44.485020 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":294,"threads_started":1}
I20260924 11:33:44.487640 8025 heartbeater.cc:347] Connected to a master server at 127.6.212.62:40673
I20260924 11:33:44.487915 8025 heartbeater.cc:464] Registering TS with master...
I20260924 11:33:44.488545 8025 heartbeater.cc:511] Master 127.6.212.62:40673 requested a full tablet report, sending...
I20260924 11:33:44.490300 7569 ts_manager.cc:194] Registered new tserver with Master: 2450c43689164a16b30fb7d6f95d5473 (127.6.212.3:33453)
I20260924 11:33:44.491108 7569 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.3@KRBTEST.COM'} at 127.6.212.3:41497
I20260924 11:33:44.497795 6992 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:33:44.508611 7758 heartbeater.cc:503] Master 127.6.212.62:40673 was elected leader, sending a full tablet report...
I20260924 11:33:44.518839 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:44.504200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34584 (local address 127.6.212.62:40673)
0924 11:33:44.504373 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:44.504378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:44.504938 (+ 560us) server_negotiation.cc:407] Connection header received
0924 11:33:44.505089 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:44.505093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:44.505159 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:44.505248 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:33:44.506852 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:44.507802 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:44.509286 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:44.510198 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:44.510348 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:44.513031 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:33:44.513036 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:33:44.513040 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:33:44.513075 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:33:44.516110 (+ 3035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:44.516719 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:44.516724 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:44.516727 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:44.516855 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:44.517195 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:44.517198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:44.517201 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:44.517956 (+ 755us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:33:44.518146 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:44.518412 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 11:33:44.518588 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 11:33:44.525291 7569 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:33:44.532212 7569 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:33:44.572333 8039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:44.544778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.3:33453 (local address 127.0.0.1:50082)
0924 11:33:44.563128 (+ 18350us) negotiation.cc:107] Waiting for socket to connect
0924 11:33:44.563146 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:33:44.563251 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:33:44.564390 (+ 1139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:33:44.564395 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:33:44.564418 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:33:44.565031 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:44.565047 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:44.566951 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:44.566956 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:44.569490 (+ 2534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:44.569504 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:44.571248 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:44.571254 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:44.571457 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:44.571527 (+ 70us) client_negotiation.cc:769] Sending connection context
0924 11:33:44.571650 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 11:33:44.571928 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18236,"mutex_wait_us":2,"thread_start_us":18117,"threads_started":1}
I20260924 11:33:44.572422 8037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:44.544772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48100 (local address 127.6.212.2:41079)
0924 11:33:44.560901 (+ 16129us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:44.560915 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:44.562713 (+ 1798us) server_negotiation.cc:407] Connection header received
0924 11:33:44.562948 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:44.562956 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:44.563254 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:44.563504 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:33:44.564417 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:44.566074 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:44.568953 (+ 2879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:44.570997 (+ 2044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:44.571128 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:44.571346 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:44.571905 (+ 559us) server_negotiation.cc:299] Negotiation successful
0924 11:33:44.572075 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15891,"thread_start_us":16154,"threads_started":1}
I20260924 11:33:44.573163 8040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:44.544735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.2:41079 (local address 127.0.0.1:48100)
0924 11:33:44.562382 (+ 17647us) negotiation.cc:107] Waiting for socket to connect
0924 11:33:44.562424 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 11:33:44.562639 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:33:44.563787 (+ 1148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:33:44.563794 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:33:44.563827 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:33:44.564212 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:44.564228 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:44.566212 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:44.566216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:44.568782 (+ 2566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:44.568797 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:44.571508 (+ 2711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:44.571518 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:44.571702 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:44.571732 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 11:33:44.571830 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 11:33:44.571937 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17379,"mutex_wait_us":117,"thread_start_us":20099,"threads_started":1}
I20260924 11:33:44.573427 8041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:44.545380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50082 (local address 127.6.212.3:33453)
0924 11:33:44.563802 (+ 18422us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:44.563810 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:44.563854 (+ 44us) server_negotiation.cc:407] Connection header received
0924 11:33:44.563955 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:44.563965 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:44.564185 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:44.564987 (+ 802us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:33:44.565304 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:44.566808 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:44.569659 (+ 2851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:44.570997 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:44.572059 (+ 1062us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:44.572272 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:44.572368 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 11:33:44.572993 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18240,"thread_start_us":18962,"threads_started":1}
I20260924 11:33:44.574328 8042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:44.545739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36956 (local address 127.6.212.1:36413)
0924 11:33:44.561696 (+ 15957us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:44.561708 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:44.566566 (+ 4858us) server_negotiation.cc:407] Connection header received
0924 11:33:44.566835 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:44.566844 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:44.567084 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:44.567310 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:33:44.568264 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:44.569478 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:44.571757 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:44.573188 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:44.573348 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:44.573570 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:44.573869 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 11:33:44.574034 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15763,"thread_start_us":15612,"threads_started":1}
I20260924 11:33:44.574221 8038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:44.545202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:36413 (local address 127.0.0.1:36956)
0924 11:33:44.566450 (+ 21248us) negotiation.cc:107] Waiting for socket to connect
0924 11:33:44.566467 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:33:44.566660 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:33:44.567597 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:33:44.567601 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:33:44.567624 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:33:44.568088 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:44.568098 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:44.569623 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:44.569630 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:44.571615 (+ 1985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:44.571625 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:44.573449 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:44.573456 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:44.573693 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:44.573714 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 11:33:44.573812 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 11:33:44.573929 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21092,"thread_start_us":16688,"threads_started":1}
I20260924 11:33:44.576962 7827 tablet_service.cc:1467] Processing CreateTablet for tablet 064e0dba01ae45aea6662f88b7751b21 (DEFAULT_TABLE table=test-table [id=eef17cfa99754070bcb459fb8d66c1b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:33:44.577032 7960 tablet_service.cc:1467] Processing CreateTablet for tablet 064e0dba01ae45aea6662f88b7751b21 (DEFAULT_TABLE table=test-table [id=eef17cfa99754070bcb459fb8d66c1b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:33:44.578208 7827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 064e0dba01ae45aea6662f88b7751b21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:33:44.578405 7960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 064e0dba01ae45aea6662f88b7751b21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:33:44.578811 7693 tablet_service.cc:1467] Processing CreateTablet for tablet 064e0dba01ae45aea6662f88b7751b21 (DEFAULT_TABLE table=test-table [id=eef17cfa99754070bcb459fb8d66c1b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:33:44.580076 7693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 064e0dba01ae45aea6662f88b7751b21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:33:44.591467 8043 tablet_bootstrap.cc:492] T 064e0dba01ae45aea6662f88b7751b21 P c33ef45fa8e94450841709e411f7ece8: Bootstrap starting.
I20260924 11:33:44.594816 8043 tablet_bootstrap.cc:654] T 064e0dba01ae45aea6662f88b7751b21 P c33ef45fa8e94450841709e411f7ece8: Neither blocks nor log segments found. Creating new log.
I20260924 11:33:44.596163 8043 log.cc:824] T 064e0dba01ae45aea6662f88b7751b21 P c33ef45fa8e94450841709e411f7ece8: Log is configured to *not* fsync() on all Append() calls
I20260924 11:33:44.596825 8044 tablet_bootstrap.cc:492] T 064e0dba01ae45aea6662f88b7751b21 P 2450c43689164a16b30fb7d6f95d5473: Bootstrap starting.
I20260924 11:33:44.597059 8045 tablet_bootstrap.cc:492] T 064e0dba01ae45aea6662f88b7751b21 P 5a81749df1664cd8999b5cabe1cfd247: Bootstrap starting.
I20260924 11:33:44.601320 8043 tablet_bootstrap.cc:492] T 064e0dba01ae45aea6662f88b7751b21 P c33ef45fa8e94450841709e411f7ece8: No bootstrap required, opened a new log
I20260924 11:33:44.601351 8045 tablet_bootstrap.cc:654] T 064e0dba01ae45aea6662f88b7751b21 P 5a81749df1664cd8999b5cabe1cfd247: Neither blocks nor log segments found. Creating new log.
I20260924 11:33:44.601353 8044 tablet_bootstrap.cc:654] T 064e0dba01ae45aea6662f88b7751b21 P 2450c43689164a16b30fb7d6f95d5473: Neither blocks nor log segments found. Creating new log.
I20260924 11:33:44.601634 8043 ts_tablet_manager.cc:1397] T 064e0dba01ae45aea6662f88b7751b21 P c33ef45fa8e94450841709e411f7ece8: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260924 11:33:44.602608 8045 log.cc:824] T 064e0dba01ae45aea6662f88b7751b21 P 5a81749df1664cd8999b5cabe1cfd247: Log is configured to *not* fsync() on all Append() calls
I20260924 11:33:44.602608 8044 log.cc:824] T 064e0dba01ae45aea6662f88b7751b21 P 2450c43689164a16b30fb7d6f95d5473: Log is configured to *not* fsync() on all Append() calls
I20260924 11:33:44.605513 8045 tablet_bootstrap.cc:492] T 064e0dba01ae45aea6662f88b7751b21 P 5a81749df1664cd8999b5cabe1cfd247: No bootstrap required, opened a new log
I20260924 11:33:44.605799 8045 ts_tablet_manager.cc:1397] T 064e0dba01ae45aea6662f88b7751b21 P 5a81749df1664cd8999b5cabe1cfd247: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260924 11:33:44.607478 8044 tablet_bootstrap.cc:492] T 064e0dba01ae45aea6662f88b7751b21 P 2450c43689164a16b30fb7d6f95d5473: No bootstrap required, opened a new log
I20260924 11:33:44.607730 8044 ts_tablet_manager.cc:1397] T 064e0dba01ae45aea6662f88b7751b21 P 2450c43689164a16b30fb7d6f95d5473: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20260924 11:33:44.612412 8043 raft_consensus.cc:348] T 064e0dba01ae45aea6662f88b7751b21 P c33ef45fa8e94450841709e411f7ece8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33ef45fa8e94450841709e411f7ece8" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 41079 } } peers { permanent_uuid: "2450c43689164a16b30fb7d6f95d5473" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 33453 } } peers { permanent_uuid: "5a81749df1664cd8999b5cabe1cfd247" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 36413 } }
I20260924 11:33:44.612965 8043 raft_consensus.cc:374] T 064e0dba01ae45aea6662f88b7751b21 P c33ef45fa8e94450841709e411f7ece8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:33:44.613089 8043 raft_consensus.cc:729] T 064e0dba01ae45aea6662f88b7751b21 P c33ef45fa8e94450841709e411f7ece8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c33ef45fa8e94450841709e411f7ece8, State: Initialized, Role: FOLLOWER
I20260924 11:33:44.613960 8043 consensus_queue.cc:260] T 064e0dba01ae45aea6662f88b7751b21 P c33ef45fa8e94450841709e411f7ece8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33ef45fa8e94450841709e411f7ece8" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 41079 } } peers { permanent_uuid: "2450c43689164a16b30fb7d6f95d5473" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 33453 } } peers { permanent_uuid: "5a81749df1664cd8999b5cabe1cfd247" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 36413 } }
I20260924 11:33:44.616232 7892 heartbeater.cc:503] Master 127.6.212.62:40673 was elected leader, sending a full tablet report...
I20260924 11:33:44.617204 8043 ts_tablet_manager.cc:1428] T 064e0dba01ae45aea6662f88b7751b21 P c33ef45fa8e94450841709e411f7ece8: Time spent starting tablet: real 0.015s user 0.010s sys 0.005s
I20260924 11:33:44.617187 8045 raft_consensus.cc:348] T 064e0dba01ae45aea6662f88b7751b21 P 5a81749df1664cd8999b5cabe1cfd247 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33ef45fa8e94450841709e411f7ece8" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 41079 } } peers { permanent_uuid: "2450c43689164a16b30fb7d6f95d5473" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 33453 } } peers { permanent_uuid: "5a81749df1664cd8999b5cabe1cfd247" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 36413 } }
I20260924 11:33:44.617607 8045 raft_consensus.cc:374] T 064e0dba01ae45aea6662f88b7751b21 P 5a81749df1664cd8999b5cabe1cfd247 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:33:44.617690 8045 raft_consensus.cc:729] T 064e0dba01ae45aea6662f88b7751b21 P 5a81749df1664cd8999b5cabe1cfd247 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a81749df1664cd8999b5cabe1cfd247, State: Initialized, Role: FOLLOWER
I20260924 11:33:44.618366 8045 consensus_queue.cc:260] T 064e0dba01ae45aea6662f88b7751b21 P 5a81749df1664cd8999b5cabe1cfd247 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33ef45fa8e94450841709e411f7ece8" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 41079 } } peers { permanent_uuid: "2450c43689164a16b30fb7d6f95d5473" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 33453 } } peers { permanent_uuid: "5a81749df1664cd8999b5cabe1cfd247" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 36413 } }
I20260924 11:33:44.618719 8044 raft_consensus.cc:348] T 064e0dba01ae45aea6662f88b7751b21 P 2450c43689164a16b30fb7d6f95d5473 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33ef45fa8e94450841709e411f7ece8" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 41079 } } peers { permanent_uuid: "2450c43689164a16b30fb7d6f95d5473" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 33453 } } peers { permanent_uuid: "5a81749df1664cd8999b5cabe1cfd247" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 36413 } }
I20260924 11:33:44.619231 8044 raft_consensus.cc:374] T 064e0dba01ae45aea6662f88b7751b21 P 2450c43689164a16b30fb7d6f95d5473 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:33:44.619341 8044 raft_consensus.cc:729] T 064e0dba01ae45aea6662f88b7751b21 P 2450c43689164a16b30fb7d6f95d5473 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2450c43689164a16b30fb7d6f95d5473, State: Initialized, Role: FOLLOWER
I20260924 11:33:44.620096 8044 consensus_queue.cc:260] T 064e0dba01ae45aea6662f88b7751b21 P 2450c43689164a16b30fb7d6f95d5473 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33ef45fa8e94450841709e411f7ece8" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 41079 } } peers { permanent_uuid: "2450c43689164a16b30fb7d6f95d5473" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 33453 } } peers { permanent_uuid: "5a81749df1664cd8999b5cabe1cfd247" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 36413 } }
I20260924 11:33:44.620997 8045 ts_tablet_manager.cc:1428] T 064e0dba01ae45aea6662f88b7751b21 P 5a81749df1664cd8999b5cabe1cfd247: Time spent starting tablet: real 0.015s user 0.010s sys 0.006s
I20260924 11:33:44.622308 8025 heartbeater.cc:503] Master 127.6.212.62:40673 was elected leader, sending a full tablet report...
I20260924 11:33:44.623034 8044 ts_tablet_manager.cc:1428] T 064e0dba01ae45aea6662f88b7751b21 P 2450c43689164a16b30fb7d6f95d5473: Time spent starting tablet: real 0.015s user 0.009s sys 0.005s
W20260924 11:33:44.713578 8026 tablet.cc:2369] T 064e0dba01ae45aea6662f88b7751b21 P 2450c43689164a16b30fb7d6f95d5473: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:33:44.722709 7893 tablet.cc:2369] T 064e0dba01ae45aea6662f88b7751b21 P c33ef45fa8e94450841709e411f7ece8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:33:44.727573 7759 tablet.cc:2369] T 064e0dba01ae45aea6662f88b7751b21 P 5a81749df1664cd8999b5cabe1cfd247: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:33:44.838178 8049 raft_consensus.cc:482] T 064e0dba01ae45aea6662f88b7751b21 P c33ef45fa8e94450841709e411f7ece8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:33:44.838523 8049 raft_consensus.cc:504] T 064e0dba01ae45aea6662f88b7751b21 P c33ef45fa8e94450841709e411f7ece8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33ef45fa8e94450841709e411f7ece8" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 41079 } } peers { permanent_uuid: "2450c43689164a16b30fb7d6f95d5473" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 33453 } } peers { permanent_uuid: "5a81749df1664cd8999b5cabe1cfd247" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 36413 } }
I20260924 11:33:44.840338 8049 leader_election.cc:288] T 064e0dba01ae45aea6662f88b7751b21 P c33ef45fa8e94450841709e411f7ece8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2450c43689164a16b30fb7d6f95d5473 (127.6.212.3:33453), 5a81749df1664cd8999b5cabe1cfd247 (127.6.212.1:36413)
I20260924 11:33:44.851348 8041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:44.840913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:59915 (local address 127.6.212.3:33453)
0924 11:33:44.841116 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:44.841123 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:44.841856 (+ 733us) server_negotiation.cc:407] Connection header received
0924 11:33:44.842063 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:44.842069 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:44.842147 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:44.842246 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:33:44.843277 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:44.844714 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:44.847991 (+ 3277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:44.849523 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:44.849663 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:44.849775 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:44.850841 (+ 1066us) server_negotiation.cc:299] Negotiation successful
0924 11:33:44.850995 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 11:33:44.851325 8042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:44.840916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:38687 (local address 127.6.212.1:36413)
0924 11:33:44.841116 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:44.841123 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:44.841857 (+ 734us) server_negotiation.cc:407] Connection header received
0924 11:33:44.842063 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:44.842070 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:44.842147 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:44.842246 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:33:44.843277 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:44.844719 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:44.847990 (+ 3271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:44.849442 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:44.849632 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:44.849769 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:44.850859 (+ 1090us) server_negotiation.cc:299] Negotiation successful
0924 11:33:44.851001 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 11:33:44.851496 8052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:44.840725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.3:33453 (local address 127.6.212.2:59915)
0924 11:33:44.841729 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0924 11:33:44.841742 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 11:33:44.841894 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:33:44.842312 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:33:44.842326 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:33:44.842377 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:33:44.843054 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:44.843069 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:44.844889 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:44.844895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:44.847795 (+ 2900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:44.847809 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:44.850305 (+ 2496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:44.850319 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:44.850605 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:44.850661 (+ 56us) client_negotiation.cc:769] Sending connection context
0924 11:33:44.850764 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 11:33:44.850951 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":911,"mutex_wait_us":260,"spinlock_wait_cycles":64384,"thread_start_us":572,"threads_started":1}
I20260924 11:33:44.851894 8053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:44.840650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:36413 (local address 127.6.212.2:38687)
0924 11:33:44.841706 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0924 11:33:44.841731 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 11:33:44.841889 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:33:44.842312 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:33:44.842317 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:33:44.842376 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:33:44.843059 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:44.843069 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:44.844890 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:44.844895 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:44.847794 (+ 2899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:44.847809 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:44.850330 (+ 2521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:44.850334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:44.850551 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:44.850661 (+ 110us) client_negotiation.cc:769] Sending connection context
0924 11:33:44.850764 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 11:33:44.850950 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":923,"mutex_wait_us":350,"spinlock_wait_cycles":14336,"thread_start_us":798,"threads_started":1}
I20260924 11:33:44.852850 7980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "064e0dba01ae45aea6662f88b7751b21" candidate_uuid: "c33ef45fa8e94450841709e411f7ece8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2450c43689164a16b30fb7d6f95d5473" is_pre_election: true
I20260924 11:33:44.853464 7980 raft_consensus.cc:2454] T 064e0dba01ae45aea6662f88b7751b21 P 2450c43689164a16b30fb7d6f95d5473 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c33ef45fa8e94450841709e411f7ece8 in term 0.
I20260924 11:33:44.854192 7783 leader_election.cc:302] T 064e0dba01ae45aea6662f88b7751b21 P c33ef45fa8e94450841709e411f7ece8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2450c43689164a16b30fb7d6f95d5473, c33ef45fa8e94450841709e411f7ece8; no voters:
I20260924 11:33:44.854645 8049 raft_consensus.cc:2781] T 064e0dba01ae45aea6662f88b7751b21 P c33ef45fa8e94450841709e411f7ece8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:33:44.854800 8049 raft_consensus.cc:482] T 064e0dba01ae45aea6662f88b7751b21 P c33ef45fa8e94450841709e411f7ece8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:33:44.854904 8049 raft_consensus.cc:3037] T 064e0dba01ae45aea6662f88b7751b21 P c33ef45fa8e94450841709e411f7ece8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:33:44.854864 7713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "064e0dba01ae45aea6662f88b7751b21" candidate_uuid: "c33ef45fa8e94450841709e411f7ece8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a81749df1664cd8999b5cabe1cfd247" is_pre_election: true
I20260924 11:33:44.855489 7713 raft_consensus.cc:2454] T 064e0dba01ae45aea6662f88b7751b21 P 5a81749df1664cd8999b5cabe1cfd247 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c33ef45fa8e94450841709e411f7ece8 in term 0.
I20260924 11:33:44.857502 8049 raft_consensus.cc:504] T 064e0dba01ae45aea6662f88b7751b21 P c33ef45fa8e94450841709e411f7ece8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33ef45fa8e94450841709e411f7ece8" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 41079 } } peers { permanent_uuid: "2450c43689164a16b30fb7d6f95d5473" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 33453 } } peers { permanent_uuid: "5a81749df1664cd8999b5cabe1cfd247" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 36413 } }
I20260924 11:33:44.858150 8049 leader_election.cc:288] T 064e0dba01ae45aea6662f88b7751b21 P c33ef45fa8e94450841709e411f7ece8 [CANDIDATE]: Term 1 election: Requested vote from peers 2450c43689164a16b30fb7d6f95d5473 (127.6.212.3:33453), 5a81749df1664cd8999b5cabe1cfd247 (127.6.212.1:36413)
I20260924 11:33:44.858505 7980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "064e0dba01ae45aea6662f88b7751b21" candidate_uuid: "c33ef45fa8e94450841709e411f7ece8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2450c43689164a16b30fb7d6f95d5473"
I20260924 11:33:44.858726 7980 raft_consensus.cc:3037] T 064e0dba01ae45aea6662f88b7751b21 P 2450c43689164a16b30fb7d6f95d5473 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:33:44.858968 7713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "064e0dba01ae45aea6662f88b7751b21" candidate_uuid: "c33ef45fa8e94450841709e411f7ece8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a81749df1664cd8999b5cabe1cfd247"
I20260924 11:33:44.859200 7713 raft_consensus.cc:3037] T 064e0dba01ae45aea6662f88b7751b21 P 5a81749df1664cd8999b5cabe1cfd247 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:33:44.861202 7980 raft_consensus.cc:2454] T 064e0dba01ae45aea6662f88b7751b21 P 2450c43689164a16b30fb7d6f95d5473 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c33ef45fa8e94450841709e411f7ece8 in term 1.
I20260924 11:33:44.861572 7783 leader_election.cc:302] T 064e0dba01ae45aea6662f88b7751b21 P c33ef45fa8e94450841709e411f7ece8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2450c43689164a16b30fb7d6f95d5473, c33ef45fa8e94450841709e411f7ece8; no voters:
I20260924 11:33:44.861824 8049 raft_consensus.cc:2781] T 064e0dba01ae45aea6662f88b7751b21 P c33ef45fa8e94450841709e411f7ece8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:33:44.863399 7713 raft_consensus.cc:2454] T 064e0dba01ae45aea6662f88b7751b21 P 5a81749df1664cd8999b5cabe1cfd247 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c33ef45fa8e94450841709e411f7ece8 in term 1.
I20260924 11:33:44.863755 8049 raft_consensus.cc:686] T 064e0dba01ae45aea6662f88b7751b21 P c33ef45fa8e94450841709e411f7ece8 [term 1 LEADER]: Becoming Leader. State: Replica: c33ef45fa8e94450841709e411f7ece8, State: Running, Role: LEADER
I20260924 11:33:44.864328 8049 consensus_queue.cc:237] T 064e0dba01ae45aea6662f88b7751b21 P c33ef45fa8e94450841709e411f7ece8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33ef45fa8e94450841709e411f7ece8" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 41079 } } peers { permanent_uuid: "2450c43689164a16b30fb7d6f95d5473" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 33453 } } peers { permanent_uuid: "5a81749df1664cd8999b5cabe1cfd247" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 36413 } }
I20260924 11:33:44.868955 7568 catalog_manager.cc:5495] T 064e0dba01ae45aea6662f88b7751b21 P c33ef45fa8e94450841709e411f7ece8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c33ef45fa8e94450841709e411f7ece8 (127.6.212.2). New cstate: current_term: 1 leader_uuid: "c33ef45fa8e94450841709e411f7ece8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33ef45fa8e94450841709e411f7ece8" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 41079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2450c43689164a16b30fb7d6f95d5473" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 33453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a81749df1664cd8999b5cabe1cfd247" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 36413 } health_report { overall_health: UNKNOWN } } }
Sep 24 11:33:44 dist-test-slave-cq7c krb5kdc[7524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.1@KRBTEST.COM
I20260924 11:33:44.954413 8042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:44.938399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36958 (local address 127.6.212.1:36413)
0924 11:33:44.938651 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:44.938659 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:44.939149 (+ 490us) server_negotiation.cc:407] Connection header received
0924 11:33:44.939387 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:44.939391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:44.939470 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:44.939582 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:33:44.941118 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:44.941940 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:44.943678 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:44.944100 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:44.944235 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:44.948381 (+ 4146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:33:44.948399 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:33:44.948416 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:33:44.948459 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:33:44.951562 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:44.952303 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:44.952320 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:44.952323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:44.952498 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:44.952975 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:44.952979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:44.952981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:44.953322 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:33:44.953563 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:44.953894 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 11:33:44.954080 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
Sep 24 11:33:44 dist-test-slave-cq7c krb5kdc[7524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249624, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 11:33:44 dist-test-slave-cq7c krb5kdc[7524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249624, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.212.1@KRBTEST.COM
I20260924 11:33:45.004149 8042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:44.988985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36974 (local address 127.6.212.1:36413)
0924 11:33:44.989214 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:44.989223 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:44.989606 (+ 383us) server_negotiation.cc:407] Connection header received
0924 11:33:44.989826 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:44.989836 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:44.989898 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:44.990007 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:33:44.991298 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:44.992110 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:44.993470 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:44.993905 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:44.994046 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:44.997912 (+ 3866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:33:44.997917 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:33:44.997920 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:33:44.997951 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:33:45.001015 (+ 3064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:45.001792 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:45.001811 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:45.001816 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:45.001962 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:45.002430 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:45.002438 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:45.002442 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:45.002744 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:33:45.003036 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:45.003433 (+ 397us) server_negotiation.cc:299] Negotiation successful
0924 11:33:45.003709 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Sep 24 11:33:45 dist-test-slave-cq7c krb5kdc[7524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 11:33:45 dist-test-slave-cq7c krb5kdc[7524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.1@KRBTEST.COM
I20260924 11:33:45.048728 8042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:45.035185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36988 (local address 127.6.212.1:36413)
0924 11:33:45.035450 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:45.035457 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:45.035659 (+ 202us) server_negotiation.cc:407] Connection header received
0924 11:33:45.035864 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:45.035870 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:45.035941 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:45.036055 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:33:45.037285 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:45.037980 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:45.039290 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:45.039794 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:45.039935 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:45.043970 (+ 4035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:33:45.043988 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:33:45.043992 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:33:45.044022 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:33:45.046645 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:45.047229 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:45.047232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:45.047235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:45.047340 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:45.047677 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:45.047680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:45.047682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:45.047942 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:33:45.048131 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:45.048313 (+ 182us) server_negotiation.cc:299] Negotiation successful
0924 11:33:45.048471 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260924 11:33:45.050462 7673 tablet_service.cc:2908] Rejecting scan request for tablet 064e0dba01ae45aea6662f88b7751b21: Uninitialized: clean time has not yet been initialized
I20260924 11:33:45.053086 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 7630
I20260924 11:33:45.065394 8042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:45.053922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36994 (local address 127.6.212.1:36413)
0924 11:33:45.054098 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:45.054103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:45.054539 (+ 436us) server_negotiation.cc:407] Connection header received
0924 11:33:45.054637 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:45.054640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:45.054702 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:45.054776 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:33:45.055910 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:45.056579 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:45.057691 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:45.058088 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:45.058181 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:45.060392 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:33:45.060397 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:33:45.060400 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:33:45.060434 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:33:45.063318 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:45.063945 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:45.063948 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:45.063951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:45.064065 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:45.064398 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:45.064401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:45.064403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:45.064627 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:33:45.064841 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:45.065018 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 11:33:45.065167 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 11:33:45.066001 7753 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:33:45.157145 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 7630
I20260924 11:33:45.173744 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 7764
Sep 24 11:33:45 dist-test-slave-cq7c krb5kdc[7524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.2@KRBTEST.COM
I20260924 11:33:45.189182 8068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:45.174587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48116 (local address 127.6.212.2:41079)
0924 11:33:45.175051 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:45.175055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:45.175070 (+ 15us) server_negotiation.cc:407] Connection header received
0924 11:33:45.175124 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:45.175127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:45.175201 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:45.175295 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:33:45.176563 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:45.177257 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:45.178701 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:45.179076 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:45.179175 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:45.183310 (+ 4135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:33:45.183325 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:33:45.183342 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:33:45.183440 (+ 98us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:33:45.186239 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:45.186921 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:45.186928 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:45.186930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:45.187066 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:45.187673 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:45.187677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:45.187679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:45.188017 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:33:45.188307 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:45.188619 (+ 312us) server_negotiation.cc:299] Negotiation successful
0924 11:33:45.188831 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":272,"threads_started":1}
I20260924 11:33:45.189816 7887 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:33:45.282823 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 7764
I20260924 11:33:45.299947 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 7897
Sep 24 11:33:45 dist-test-slave-cq7c krb5kdc[7524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.3@KRBTEST.COM
I20260924 11:33:45.315238 8041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:45.300736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50084 (local address 127.6.212.3:33453)
0924 11:33:45.300895 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:45.300899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:45.300917 (+ 18us) server_negotiation.cc:407] Connection header received
0924 11:33:45.301047 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:45.301051 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:45.301122 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:45.301203 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:33:45.302488 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:45.303201 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:45.304628 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:45.305031 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:45.305127 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:45.309278 (+ 4151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:33:45.309294 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:33:45.309309 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:33:45.309343 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:33:45.312314 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:45.313007 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:45.313015 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:45.313036 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:45.313204 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:45.313580 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:45.313586 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:45.313589 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:45.314003 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:33:45.314279 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:45.314592 (+ 313us) server_negotiation.cc:299] Negotiation successful
0924 11:33:45.314823 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 11:33:45.315953 8020 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:33:45.402534 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 7897
I20260924 11:33:45.418576 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 7537
I20260924 11:33:45.419811 7599 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:33:45.526736 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 7537
2026-09-24T11:33:45Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4376 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:33:45 dist-test-slave-cq7c krb5kdc[8074](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:33:45 dist-test-slave-cq7c krb5kdc[8074](info): set up 2 sockets
Sep 24 11:33:45 dist-test-slave-cq7c krb5kdc[8074](info): commencing operation
krb5kdc: starting...
W20260924 11:33:46.627310 6992 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Sep 24 11:33:46 dist-test-slave-cq7c krb5kdc[8074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:33:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:33:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.212.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
I20260924 11:33:46.828068 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:40625
--webserver_interface=127.6.212.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:42785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:40625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:33:47.154016 8087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:33:47.154358 8087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:33:47.154518 8087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:33:47.161957 8087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:33:47.162102 8087 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 11:33:47.162175 8087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:33:47.162266 8087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:33:47.162371 8087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:33:47.162472 8087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:33:47.172430 8087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:42785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:40625
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:40625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.212.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:33:47.173744 8087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:33:47.175182 8087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:33:47.187057 8093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:33:47.187088 8092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:33:47.187224 8095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:33:47.187856 8087 server_base.cc:1034] running on GCE node
I20260924 11:33:47.188647 8087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:33:47.190038 8087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:33:47.191269 8087 hybrid_clock.cc:648] HybridClock initialized: now 1790249627191199 us; error 41 us; skew 500 ppm
I20260924 11:33:47.191759 8087 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:33:47 dist-test-slave-cq7c krb5kdc[8074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249627, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:33:47.195025 8087 init.cc:377] Logged in from keytab as kudu/127.6.212.62@KRBTEST.COM (short username kudu)
I20260924 11:33:47.196463 8087 webserver.cc:460] Webserver started at http://127.6.212.62:40441/ using document root <none> and password file <none>
I20260924 11:33:47.197148 8087 fs_manager.cc:359] Metadata directory not provided
I20260924 11:33:47.197232 8087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:33:47.197511 8087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:33:47.200452 8087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "092352e3835147d59f8e8772a70a9475"
format_stamp: "Formatted at 2026-09-24 11:33:47 on dist-test-slave-cq7c"
server_key: "e6879c61283e4e8a2b2a260396b19430"
server_key_iv: "a251963cfbc6d51d76fffc1255b8d40b"
server_key_version: "encryptionkey@0"
I20260924 11:33:47.201182 8087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "092352e3835147d59f8e8772a70a9475"
format_stamp: "Formatted at 2026-09-24 11:33:47 on dist-test-slave-cq7c"
server_key: "e6879c61283e4e8a2b2a260396b19430"
server_key_iv: "a251963cfbc6d51d76fffc1255b8d40b"
server_key_version: "encryptionkey@0"
I20260924 11:33:47.206277 8087 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260924 11:33:47.209725 8102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:33:47.211107 8087 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 11:33:47.211305 8087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "092352e3835147d59f8e8772a70a9475"
format_stamp: "Formatted at 2026-09-24 11:33:47 on dist-test-slave-cq7c"
server_key: "e6879c61283e4e8a2b2a260396b19430"
server_key_iv: "a251963cfbc6d51d76fffc1255b8d40b"
server_key_version: "encryptionkey@0"
I20260924 11:33:47.211607 8087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:33:47.245676 8087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:33:47.250203 8087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:33:47.250490 8087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:33:47.272581 8087 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.62:40625
I20260924 11:33:47.272600 8163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.62:40625 every 8 connection(s)
I20260924 11:33:47.273926 8087 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:33:47.275197 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 8087
I20260924 11:33:47.275444 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:33:47.276226 6992 external_mini_cluster.cc:1428] Setting key ccadb64b021464a001000c29bc9bbe1a
I20260924 11:33:47.278915 8164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 11:33:47 dist-test-slave-cq7c krb5kdc[8074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:33:47.291987 8164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 092352e3835147d59f8e8772a70a9475: Bootstrap starting.
I20260924 11:33:47.296165 8164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 092352e3835147d59f8e8772a70a9475: Neither blocks nor log segments found. Creating new log.
I20260924 11:33:47.297497 8164 log.cc:824] T 00000000000000000000000000000000 P 092352e3835147d59f8e8772a70a9475: Log is configured to *not* fsync() on all Append() calls
I20260924 11:33:47.300164 8167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:47.278329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52786 (local address 127.6.212.62:40625)
0924 11:33:47.279119 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:47.279130 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:47.279170 (+ 40us) server_negotiation.cc:407] Connection header received
0924 11:33:47.280769 (+ 1599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:47.280789 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:47.281161 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:47.281491 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:33:47.283254 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:47.284719 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:47.286456 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:47.287107 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:47.287251 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:47.291898 (+ 4647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:33:47.291908 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:33:47.291926 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:33:47.291991 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:33:47.295668 (+ 3677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:47.296300 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:47.296310 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:47.296320 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:47.296541 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:47.296896 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:47.296901 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:47.296903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:47.297621 (+ 718us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:33:47.297904 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:47.298369 (+ 465us) server_negotiation.cc:299] Negotiation successful
0924 11:33:47.298789 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":360,"threads_started":1}
I20260924 11:33:47.301015 8164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 092352e3835147d59f8e8772a70a9475: No bootstrap required, opened a new log
I20260924 11:33:47.308912 8164 raft_consensus.cc:348] T 00000000000000000000000000000000 P 092352e3835147d59f8e8772a70a9475 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "092352e3835147d59f8e8772a70a9475" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 40625 } }
I20260924 11:33:47.309254 8164 raft_consensus.cc:374] T 00000000000000000000000000000000 P 092352e3835147d59f8e8772a70a9475 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:33:47.309330 8164 raft_consensus.cc:729] T 00000000000000000000000000000000 P 092352e3835147d59f8e8772a70a9475 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 092352e3835147d59f8e8772a70a9475, State: Initialized, Role: FOLLOWER
I20260924 11:33:47.310036 8164 consensus_queue.cc:260] T 00000000000000000000000000000000 P 092352e3835147d59f8e8772a70a9475 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "092352e3835147d59f8e8772a70a9475" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 40625 } }
I20260924 11:33:47.310251 8164 raft_consensus.cc:388] T 00000000000000000000000000000000 P 092352e3835147d59f8e8772a70a9475 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:33:47.310354 8164 raft_consensus.cc:482] T 00000000000000000000000000000000 P 092352e3835147d59f8e8772a70a9475 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:33:47.310489 8164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 092352e3835147d59f8e8772a70a9475 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:33:47.312949 8164 raft_consensus.cc:504] T 00000000000000000000000000000000 P 092352e3835147d59f8e8772a70a9475 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "092352e3835147d59f8e8772a70a9475" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 40625 } }
I20260924 11:33:47.313431 8164 leader_election.cc:302] T 00000000000000000000000000000000 P 092352e3835147d59f8e8772a70a9475 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 092352e3835147d59f8e8772a70a9475; no voters:
I20260924 11:33:47.313961 8164 leader_election.cc:288] T 00000000000000000000000000000000 P 092352e3835147d59f8e8772a70a9475 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:33:47.314108 8169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 092352e3835147d59f8e8772a70a9475 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:33:47.314927 8169 raft_consensus.cc:686] T 00000000000000000000000000000000 P 092352e3835147d59f8e8772a70a9475 [term 1 LEADER]: Becoming Leader. State: Replica: 092352e3835147d59f8e8772a70a9475, State: Running, Role: LEADER
I20260924 11:33:47.315493 8169 consensus_queue.cc:237] T 00000000000000000000000000000000 P 092352e3835147d59f8e8772a70a9475 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "092352e3835147d59f8e8772a70a9475" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 40625 } }
I20260924 11:33:47.316432 8164 sys_catalog.cc:564] T 00000000000000000000000000000000 P 092352e3835147d59f8e8772a70a9475 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:33:47.318776 8170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 092352e3835147d59f8e8772a70a9475 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "092352e3835147d59f8e8772a70a9475" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "092352e3835147d59f8e8772a70a9475" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 40625 } } }
I20260924 11:33:47.319270 8170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 092352e3835147d59f8e8772a70a9475 [sys.catalog]: This master's current role is: LEADER
I20260924 11:33:47.318751 8171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 092352e3835147d59f8e8772a70a9475 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 092352e3835147d59f8e8772a70a9475. Latest consensus state: current_term: 1 leader_uuid: "092352e3835147d59f8e8772a70a9475" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "092352e3835147d59f8e8772a70a9475" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 40625 } } }
I20260924 11:33:47.319530 8171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 092352e3835147d59f8e8772a70a9475 [sys.catalog]: This master's current role is: LEADER
I20260924 11:33:47.320849 8177 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:33:47.325940 8177 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:33:47.335258 8177 catalog_manager.cc:1347] Generated new cluster ID: 2194319d3a3040298ceeaab94b937684
I20260924 11:33:47.335408 8177 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:33:47.394380 8177 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:33:47.396164 8177 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:33:47.414291 8177 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 092352e3835147d59f8e8772a70a9475: Generated new TSK 0
I20260924 11:33:47.415284 8177 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.212.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab.
Entry for principal kudu/127.6.212.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab.
I20260924 11:33:47.463636 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.1:0
--local_ip_for_outbound_sockets=127.6.212.1
--webserver_interface=127.6.212.1
--webserver_port=0
--tserver_master_addrs=127.6.212.62:40625
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab
--principal=kudu/127.6.212.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:42785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:33:47.785703 8190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:33:47.785982 8190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:33:47.786080 8190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:33:47.793247 8190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:33:47.793349 8190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:33:47.793445 8190 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 11:33:47.793517 8190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.1
I20260924 11:33:47.802980 8190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:42785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab
--principal=kudu/127.6.212.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.212.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.212.62:40625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.1
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:33:47.804209 8190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:33:47.805446 8190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:33:47.819006 8196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:33:47.819095 8195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:33:47.819041 8198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:33:47.820107 8190 server_base.cc:1034] running on GCE node
I20260924 11:33:47.820973 8190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:33:47.822136 8190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:33:47.823504 8190 hybrid_clock.cc:648] HybridClock initialized: now 1790249627823438 us; error 57 us; skew 500 ppm
I20260924 11:33:47.823855 8190 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:33:47 dist-test-slave-cq7c krb5kdc[8074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249627, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:33:47.833072 8190 init.cc:377] Logged in from keytab as kudu/127.6.212.1@KRBTEST.COM (short username kudu)
I20260924 11:33:47.834751 8190 webserver.cc:460] Webserver started at http://127.6.212.1:38997/ using document root <none> and password file <none>
I20260924 11:33:47.835603 8190 fs_manager.cc:359] Metadata directory not provided
I20260924 11:33:47.835744 8190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:33:47.836107 8190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:33:47.839422 8190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-0/data/instance:
uuid: "e7e9dd80d738487cb5e99a5d55eded82"
format_stamp: "Formatted at 2026-09-24 11:33:47 on dist-test-slave-cq7c"
server_key: "5c480a59aa08f532398721c556e48101"
server_key_iv: "2f3aa04b45b3d0e0370819fef6bc7d35"
server_key_version: "encryptionkey@0"
I20260924 11:33:47.840343 8190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance:
uuid: "e7e9dd80d738487cb5e99a5d55eded82"
format_stamp: "Formatted at 2026-09-24 11:33:47 on dist-test-slave-cq7c"
server_key: "5c480a59aa08f532398721c556e48101"
server_key_iv: "2f3aa04b45b3d0e0370819fef6bc7d35"
server_key_version: "encryptionkey@0"
I20260924 11:33:47.845955 8190 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 11:33:47.849725 8205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:33:47.851102 8190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 11:33:47.851290 8190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-0/wal
uuid: "e7e9dd80d738487cb5e99a5d55eded82"
format_stamp: "Formatted at 2026-09-24 11:33:47 on dist-test-slave-cq7c"
server_key: "5c480a59aa08f532398721c556e48101"
server_key_iv: "2f3aa04b45b3d0e0370819fef6bc7d35"
server_key_version: "encryptionkey@0"
I20260924 11:33:47.851583 8190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:33:47.874552 8190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:33:47.878765 8190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:33:47.879057 8190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:33:47.881897 8190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:33:47.882010 8190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:33:47.882241 8190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:33:47.882344 8190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 11:33:47 dist-test-slave-cq7c krb5kdc[8074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249627, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:33:47.905313 8213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:47.882356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:41433 (local address 127.6.212.62:40625)
0924 11:33:47.882904 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:47.882910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:47.884135 (+ 1225us) server_negotiation.cc:407] Connection header received
0924 11:33:47.886084 (+ 1949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:47.886090 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:47.886176 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:47.886323 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:33:47.888552 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:47.889829 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:47.892572 (+ 2743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:47.893351 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:47.893510 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:47.897815 (+ 4305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:33:47.897825 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:33:47.897833 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:33:47.897895 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:33:47.901843 (+ 3948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:47.902812 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:47.902820 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:47.902826 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:47.903000 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:47.903569 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:47.903574 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:47.903578 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:47.903853 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:33:47.904020 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:47.904721 (+ 701us) server_negotiation.cc:299] Negotiation successful
0924 11:33:47.904940 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":255,"threads_started":1}
I20260924 11:33:47.906319 8215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:47.883048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:40625 (local address 127.6.212.1:41433)
0924 11:33:47.883955 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0924 11:33:47.883999 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 11:33:47.885809 (+ 1810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:33:47.886695 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:33:47.886706 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:33:47.887492 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:33:47.888313 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:47.888332 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:47.890021 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:47.890029 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:47.892418 (+ 2389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:47.892432 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:47.893590 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:47.893596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:47.893771 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:47.894583 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:33:47.894630 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:33:47.897536 (+ 2906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:33:47.902037 (+ 4501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:33:47.902049 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:33:47.902072 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:33:47.902657 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:33:47.903133 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:33:47.903138 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:33:47.903141 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:33:47.903442 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:33:47.903994 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:33:47.904002 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:33:47.904370 (+ 368us) client_negotiation.cc:769] Sending connection context
0924 11:33:47.904637 (+ 267us) client_negotiation.cc:240] Negotiation successful
0924 11:33:47.905145 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":401,"threads_started":1}
I20260924 11:33:47.931288 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:47.913234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:40625 (local address 127.6.212.1:44387)
0924 11:33:47.916422 (+ 3188us) negotiation.cc:107] Waiting for socket to connect
0924 11:33:47.916437 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:33:47.916595 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:33:47.916957 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:33:47.916961 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:33:47.917852 (+ 891us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:33:47.918430 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:47.918441 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:47.919465 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:47.919473 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:47.921098 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:47.921136 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:47.922325 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:47.922335 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:47.922554 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:47.923261 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:33:47.923291 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:33:47.924354 (+ 1063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:33:47.927512 (+ 3158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:33:47.927517 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:33:47.927521 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:33:47.928160 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:33:47.928618 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:33:47.928629 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:33:47.928635 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:33:47.928839 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:33:47.929688 (+ 849us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:33:47.929696 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:33:47.929966 (+ 270us) client_negotiation.cc:769] Sending connection context
0924 11:33:47.930168 (+ 202us) client_negotiation.cc:240] Negotiation successful
0924 11:33:47.930597 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3041,"thread_start_us":2803,"threads_started":1}
I20260924 11:33:47.931859 8213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:47.913237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:44387 (local address 127.6.212.62:40625)
0924 11:33:47.913463 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:47.913471 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:47.916526 (+ 3055us) server_negotiation.cc:407] Connection header received
0924 11:33:47.916722 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:47.916727 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:47.916804 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:47.916935 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:33:47.918577 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:47.919289 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:47.921310 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:47.921867 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:47.921981 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:47.924530 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:33:47.924536 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:33:47.924541 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:33:47.924579 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:33:47.927343 (+ 2764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:47.928281 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:47.928286 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:47.928290 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:47.928481 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:47.929054 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:47.929059 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:47.929063 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:47.929438 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:33:47.929612 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:47.931188 (+ 1576us) server_negotiation.cc:299] Negotiation successful
0924 11:33:47.931446 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 11:33:47.934007 8190 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.1:46675
I20260924 11:33:47.934043 8325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.1:46675 every 8 connection(s)
I20260924 11:33:47.935331 8190 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
I20260924 11:33:47.937551 8326 heartbeater.cc:347] Connected to a master server at 127.6.212.62:40625
I20260924 11:33:47.937772 8326 heartbeater.cc:464] Registering TS with master...
I20260924 11:33:47.938390 8326 heartbeater.cc:511] Master 127.6.212.62:40625 requested a full tablet report, sending...
I20260924 11:33:47.940384 8119 ts_manager.cc:194] Registered new tserver with Master: e7e9dd80d738487cb5e99a5d55eded82 (127.6.212.1:46675)
I20260924 11:33:47.941476 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 8190
I20260924 11:33:47.941705 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance
I20260924 11:33:47.941919 8119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.1@KRBTEST.COM'} at 127.6.212.1:41433
I20260924 11:33:47.942390 6992 external_mini_cluster.cc:1428] Setting key 766220738022df1813ad0bef7cceab2b
WARNING: no policy specified for kudu/127.6.212.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab.
Entry for principal kudu/127.6.212.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab.
I20260924 11:33:47.983207 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.2:0
--local_ip_for_outbound_sockets=127.6.212.2
--webserver_interface=127.6.212.2
--webserver_port=0
--tserver_master_addrs=127.6.212.62:40625
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab
--principal=kudu/127.6.212.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:42785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:33:48.303620 8330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:33:48.303884 8330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:33:48.303988 8330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:33:48.311329 8330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:33:48.311466 8330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:33:48.311558 8330 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 11:33:48.311636 8330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.2
I20260924 11:33:48.321658 8330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:42785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab
--principal=kudu/127.6.212.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.212.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.212.62:40625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.2
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:33:48.322978 8330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:33:48.324363 8330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:33:48.338351 8338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:33:48.338310 8335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:33:48.338393 8336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:33:48.339253 8330 server_base.cc:1034] running on GCE node
I20260924 11:33:48.339920 8330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:33:48.340914 8330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:33:48.342159 8330 hybrid_clock.cc:648] HybridClock initialized: now 1790249628342096 us; error 46 us; skew 500 ppm
I20260924 11:33:48.342468 8330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:33:48 dist-test-slave-cq7c krb5kdc[8074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249628, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:33:48.345662 8330 init.cc:377] Logged in from keytab as kudu/127.6.212.2@KRBTEST.COM (short username kudu)
I20260924 11:33:48.347079 8330 webserver.cc:460] Webserver started at http://127.6.212.2:38985/ using document root <none> and password file <none>
I20260924 11:33:48.347803 8330 fs_manager.cc:359] Metadata directory not provided
I20260924 11:33:48.347891 8330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:33:48.348156 8330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:33:48.351096 8330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-1/data/instance:
uuid: "3aae29554c0f426299acdeed32fe630b"
format_stamp: "Formatted at 2026-09-24 11:33:48 on dist-test-slave-cq7c"
server_key: "1a799efb177dde1e6e70f2eeab61bd06"
server_key_iv: "819fcc107c776825d90c743bae4dd2ce"
server_key_version: "encryptionkey@0"
I20260924 11:33:48.351893 8330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance:
uuid: "3aae29554c0f426299acdeed32fe630b"
format_stamp: "Formatted at 2026-09-24 11:33:48 on dist-test-slave-cq7c"
server_key: "1a799efb177dde1e6e70f2eeab61bd06"
server_key_iv: "819fcc107c776825d90c743bae4dd2ce"
server_key_version: "encryptionkey@0"
I20260924 11:33:48.357072 8330 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 11:33:48.360440 8345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:33:48.361876 8330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 11:33:48.362056 8330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-1/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-1/wal
uuid: "3aae29554c0f426299acdeed32fe630b"
format_stamp: "Formatted at 2026-09-24 11:33:48 on dist-test-slave-cq7c"
server_key: "1a799efb177dde1e6e70f2eeab61bd06"
server_key_iv: "819fcc107c776825d90c743bae4dd2ce"
server_key_version: "encryptionkey@0"
I20260924 11:33:48.362277 8330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:33:48.373740 8330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:33:48.377512 8330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:33:48.377780 8330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:33:48.380342 8330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:33:48.380451 8330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:33:48.380649 8330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:33:48.380769 8330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 24 11:33:48 dist-test-slave-cq7c krb5kdc[8074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249628, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.2@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:33:48.404469 8213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:48.380872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:37859 (local address 127.6.212.62:40625)
0924 11:33:48.381058 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:48.381064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:48.382318 (+ 1254us) server_negotiation.cc:407] Connection header received
0924 11:33:48.384650 (+ 2332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:48.384655 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:48.384759 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:48.384856 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:33:48.387138 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:48.388220 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:48.390714 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:48.391549 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:48.391714 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:48.397010 (+ 5296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:33:48.397020 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:33:48.397028 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:33:48.397078 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:33:48.400440 (+ 3362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:48.401553 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:48.401560 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:48.401566 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:48.401734 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:48.402221 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:48.402229 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:48.402235 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:48.402687 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:33:48.402881 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:48.403622 (+ 741us) server_negotiation.cc:299] Negotiation successful
0924 11:33:48.403962 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 11:33:48.404937 8354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:48.381301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:40625 (local address 127.6.212.2:37859)
0924 11:33:48.382166 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0924 11:33:48.382220 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 11:33:48.384357 (+ 2137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:33:48.385291 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:33:48.385305 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:33:48.386096 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:33:48.386934 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:48.386978 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:48.388377 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:48.388382 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:48.390491 (+ 2109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:48.390514 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:48.391821 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:48.391831 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:48.392104 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:48.392823 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:33:48.392861 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:33:48.396704 (+ 3843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:33:48.400585 (+ 3881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:33:48.400594 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:33:48.400614 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:33:48.401384 (+ 770us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:33:48.401907 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:33:48.401911 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:33:48.401913 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:33:48.402105 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:33:48.402831 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:33:48.402836 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:33:48.403168 (+ 332us) client_negotiation.cc:769] Sending connection context
0924 11:33:48.403484 (+ 316us) client_negotiation.cc:240] Negotiation successful
0924 11:33:48.403871 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":383,"threads_started":1}
I20260924 11:33:48.424237 8213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:48.410688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:51149 (local address 127.6.212.62:40625)
0924 11:33:48.410853 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:48.410857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:48.411891 (+ 1034us) server_negotiation.cc:407] Connection header received
0924 11:33:48.412106 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:48.412112 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:48.412174 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:48.412273 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:33:48.413145 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:48.414023 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:48.415706 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:48.416223 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:48.416381 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:48.418327 (+ 1946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:33:48.418333 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:33:48.418338 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:33:48.418372 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:33:48.421278 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:48.422324 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:48.422330 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:48.422333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:48.422463 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:48.422915 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:48.422918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:48.422920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:48.423204 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:33:48.423310 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:48.423857 (+ 547us) server_negotiation.cc:299] Negotiation successful
0924 11:33:48.424016 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 11:33:48.424849 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:48.411009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:40625 (local address 127.6.212.2:51149)
0924 11:33:48.411783 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0924 11:33:48.411804 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 11:33:48.411963 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:33:48.412295 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:33:48.412299 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:33:48.412741 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:33:48.413011 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:48.413019 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:48.414150 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:48.414157 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:48.415565 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:48.415576 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:48.416429 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:48.416435 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:48.416616 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:48.417349 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:33:48.417377 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:33:48.418153 (+ 776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:33:48.421422 (+ 3269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:33:48.421430 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:33:48.421441 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:33:48.422194 (+ 753us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:33:48.422578 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:33:48.422585 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:33:48.422592 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:33:48.422786 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:33:48.423316 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:33:48.423323 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:33:48.423587 (+ 264us) client_negotiation.cc:769] Sending connection context
0924 11:33:48.423779 (+ 192us) client_negotiation.cc:240] Negotiation successful
0924 11:33:48.424229 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":419,"threads_started":1}
I20260924 11:33:48.439196 8330 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.2:36373
I20260924 11:33:48.439204 8464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.2:36373 every 8 connection(s)
I20260924 11:33:48.440526 8330 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
I20260924 11:33:48.442598 8465 heartbeater.cc:347] Connected to a master server at 127.6.212.62:40625
I20260924 11:33:48.442817 8465 heartbeater.cc:464] Registering TS with master...
I20260924 11:33:48.443511 8465 heartbeater.cc:511] Master 127.6.212.62:40625 requested a full tablet report, sending...
I20260924 11:33:48.444898 8119 ts_manager.cc:194] Registered new tserver with Master: 3aae29554c0f426299acdeed32fe630b (127.6.212.2:36373)
I20260924 11:33:48.445631 8119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.2@KRBTEST.COM'} at 127.6.212.2:37859
I20260924 11:33:48.450719 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 8330
I20260924 11:33:48.450960 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance
I20260924 11:33:48.451485 6992 external_mini_cluster.cc:1428] Setting key 3053b4d13d57f434445ad8c4814b972c
WARNING: no policy specified for kudu/127.6.212.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab.
Entry for principal kudu/127.6.212.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab.
I20260924 11:33:48.491045 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.3:0
--local_ip_for_outbound_sockets=127.6.212.3
--webserver_interface=127.6.212.3
--webserver_port=0
--tserver_master_addrs=127.6.212.62:40625
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab
--principal=kudu/127.6.212.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:42785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:33:48.808617 8469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:33:48.808880 8469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:33:48.809031 8469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:33:48.816543 8469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:33:48.816643 8469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:33:48.816818 8469 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 11:33:48.816910 8469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.3
I20260924 11:33:48.826750 8469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:42785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab
--principal=kudu/127.6.212.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.212.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.212.62:40625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.3
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:33:48.828116 8469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:33:48.829399 8469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:33:48.843019 8474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:33:48.842981 8475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:33:48.843053 8477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:33:48.843672 8469 server_base.cc:1034] running on GCE node
I20260924 11:33:48.844249 8469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:33:48.845152 8469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:33:48.846406 8469 hybrid_clock.cc:648] HybridClock initialized: now 1790249628846352 us; error 33 us; skew 500 ppm
I20260924 11:33:48.846702 8469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:33:48 dist-test-slave-cq7c krb5kdc[8074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249628, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:33:48.849916 8469 init.cc:377] Logged in from keytab as kudu/127.6.212.3@KRBTEST.COM (short username kudu)
I20260924 11:33:48.851238 8469 webserver.cc:460] Webserver started at http://127.6.212.3:43941/ using document root <none> and password file <none>
I20260924 11:33:48.851938 8469 fs_manager.cc:359] Metadata directory not provided
I20260924 11:33:48.852064 8469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:33:48.852391 8469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:33:48.855307 8469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-2/data/instance:
uuid: "c48acea63acb4d6aaf797e3574a4cad3"
format_stamp: "Formatted at 2026-09-24 11:33:48 on dist-test-slave-cq7c"
server_key: "fe5ac9db3e6d700509f1a1d1f5db48ca"
server_key_iv: "90f834cf1b80891f91cead9f5c7fd491"
server_key_version: "encryptionkey@0"
I20260924 11:33:48.856233 8469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance:
uuid: "c48acea63acb4d6aaf797e3574a4cad3"
format_stamp: "Formatted at 2026-09-24 11:33:48 on dist-test-slave-cq7c"
server_key: "fe5ac9db3e6d700509f1a1d1f5db48ca"
server_key_iv: "90f834cf1b80891f91cead9f5c7fd491"
server_key_version: "encryptionkey@0"
I20260924 11:33:48.861682 8469 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 11:33:48.865124 8484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:33:48.866464 8469 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 11:33:48.866677 8469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-2/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-2/wal
uuid: "c48acea63acb4d6aaf797e3574a4cad3"
format_stamp: "Formatted at 2026-09-24 11:33:48 on dist-test-slave-cq7c"
server_key: "fe5ac9db3e6d700509f1a1d1f5db48ca"
server_key_iv: "90f834cf1b80891f91cead9f5c7fd491"
server_key_version: "encryptionkey@0"
I20260924 11:33:48.866904 8469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:33:48.888218 8469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:33:48.892063 8469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:33:48.892342 8469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:33:48.894821 8469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:33:48.894933 8469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:33:48.895100 8469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:33:48.895176 8469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 11:33:48 dist-test-slave-cq7c krb5kdc[8074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249628, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.3@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:33:48.918918 8213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:48.895667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:47257 (local address 127.6.212.62:40625)
0924 11:33:48.895868 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:48.895874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:48.897328 (+ 1454us) server_negotiation.cc:407] Connection header received
0924 11:33:48.899347 (+ 2019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:48.899376 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:48.899484 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:48.899593 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:33:48.901667 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:48.903006 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:48.905523 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:48.906263 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:48.906430 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:48.911112 (+ 4682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:33:48.911120 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:33:48.911128 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:33:48.911176 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:33:48.915190 (+ 4014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:48.916053 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:48.916060 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:48.916064 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:48.916251 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:48.916876 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:48.916879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:48.916881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:48.917140 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:33:48.917250 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:48.918435 (+ 1185us) server_negotiation.cc:299] Negotiation successful
0924 11:33:48.918612 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 11:33:48.921042 8493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:48.896220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:40625 (local address 127.6.212.3:47257)
0924 11:33:48.897138 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0924 11:33:48.897182 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 11:33:48.899053 (+ 1871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:33:48.899855 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:33:48.899875 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:33:48.900570 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:33:48.901459 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:48.901489 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:48.903171 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:48.903175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:48.905328 (+ 2153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:48.905342 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:48.906585 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:48.906591 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:48.906755 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:48.907520 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:33:48.907592 (+ 72us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:33:48.910830 (+ 3238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:33:48.915338 (+ 4508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:33:48.915349 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:33:48.915385 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:33:48.915930 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:33:48.916410 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:33:48.916418 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:33:48.916427 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:33:48.916747 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:33:48.917281 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:33:48.917299 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:33:48.917867 (+ 568us) client_negotiation.cc:769] Sending connection context
0924 11:33:48.918258 (+ 391us) client_negotiation.cc:240] Negotiation successful
0924 11:33:48.918960 (+ 702us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":402,"threads_started":1}
I20260924 11:33:48.944967 8326 heartbeater.cc:503] Master 127.6.212.62:40625 was elected leader, sending a full tablet report...
I20260924 11:33:48.946116 8213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:48.928141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:42811 (local address 127.6.212.62:40625)
0924 11:33:48.928337 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:48.928346 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:48.930141 (+ 1795us) server_negotiation.cc:407] Connection header received
0924 11:33:48.930361 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:48.930365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:48.930428 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:48.930517 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:33:48.932147 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:48.933263 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:48.936034 (+ 2771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:48.936648 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:48.936779 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:48.939578 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:33:48.939586 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:33:48.939594 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:33:48.939670 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:33:48.942978 (+ 3308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:48.944028 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:48.944034 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:48.944040 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:48.944201 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:48.944685 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:48.944688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:48.944690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:48.944925 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:33:48.945077 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:48.945510 (+ 433us) server_negotiation.cc:299] Negotiation successful
0924 11:33:48.945724 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 11:33:48.946194 8575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:48.928730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:40625 (local address 127.6.212.3:42811)
0924 11:33:48.930006 (+ 1276us) negotiation.cc:107] Waiting for socket to connect
0924 11:33:48.930030 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 11:33:48.930206 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:33:48.930591 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:33:48.930598 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:33:48.931326 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:33:48.931946 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:48.931965 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:48.933526 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:48.933534 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:48.935863 (+ 2329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:48.935883 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:48.936875 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:48.936885 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:48.937149 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:48.937990 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:33:48.938023 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:33:48.939337 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:33:48.943125 (+ 3788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:33:48.943130 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:33:48.943134 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:33:48.943875 (+ 741us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:33:48.944347 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:33:48.944355 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:33:48.944362 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:33:48.944556 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:33:48.945048 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:33:48.945052 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:33:48.945235 (+ 183us) client_negotiation.cc:769] Sending connection context
0924 11:33:48.945392 (+ 157us) client_negotiation.cc:240] Negotiation successful
0924 11:33:48.945725 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1064,"thread_start_us":906,"threads_started":1}
I20260924 11:33:48.950253 8469 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.3:43581
I20260924 11:33:48.950271 8603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.3:43581 every 8 connection(s)
I20260924 11:33:48.951510 8469 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
I20260924 11:33:48.953322 8604 heartbeater.cc:347] Connected to a master server at 127.6.212.62:40625
I20260924 11:33:48.953516 8604 heartbeater.cc:464] Registering TS with master...
I20260924 11:33:48.954064 8604 heartbeater.cc:511] Master 127.6.212.62:40625 requested a full tablet report, sending...
I20260924 11:33:48.955250 8119 ts_manager.cc:194] Registered new tserver with Master: c48acea63acb4d6aaf797e3574a4cad3 (127.6.212.3:43581)
I20260924 11:33:48.956216 8119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.3@KRBTEST.COM'} at 127.6.212.3:47257
I20260924 11:33:48.958431 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 8469
I20260924 11:33:48.958630 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance
I20260924 11:33:48.959041 6992 external_mini_cluster.cc:1428] Setting key d470e3f114475a2f23db8bfbdff162e0
I20260924 11:33:48.961910 6992 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 11:33:48 dist-test-slave-cq7c krb5kdc[8074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249628, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 11:33:48 dist-test-slave-cq7c krb5kdc[8074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249628, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:33:49.008323 8213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:48.993678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52798 (local address 127.6.212.62:40625)
0924 11:33:48.993931 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:48.993938 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:48.994016 (+ 78us) server_negotiation.cc:407] Connection header received
0924 11:33:48.994204 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:48.994209 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:48.994289 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:48.994421 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:33:48.995800 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:48.996591 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:48.997923 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:48.998647 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:48.998830 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:49.002863 (+ 4033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:33:49.002868 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:33:49.002871 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:33:49.002907 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:33:49.005714 (+ 2807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:49.006357 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:49.006363 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:49.006367 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:49.006536 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:49.006902 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:49.006906 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:49.006908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:49.007168 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:33:49.007463 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:49.007672 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 11:33:49.007902 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 11:33:49.012612 8119 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52798:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:33:49.021014 8119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:33:49.063005 8621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:49.039880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53916 (local address 127.6.212.3:43581)
0924 11:33:49.049738 (+ 9858us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:49.049749 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:49.049792 (+ 43us) server_negotiation.cc:407] Connection header received
0924 11:33:49.049928 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:49.049936 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:49.050262 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:49.050514 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:33:49.051524 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:49.053410 (+ 1886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:49.056969 (+ 3559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:49.060159 (+ 3190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:49.060323 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:49.060671 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:49.062478 (+ 1807us) server_negotiation.cc:299] Negotiation successful
0924 11:33:49.062660 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9675,"thread_start_us":10901,"threads_started":1}
I20260924 11:33:49.063192 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:49.037290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.2:36373 (local address 127.0.0.1:54092)
0924 11:33:49.038043 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0924 11:33:49.038082 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 11:33:49.038398 (+ 316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:33:49.055667 (+ 17269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:33:49.055671 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:33:49.055695 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:33:49.056254 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:49.056265 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:49.058062 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:49.058068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:49.060515 (+ 2447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:49.060526 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:49.062345 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:49.062356 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:49.062563 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:49.062607 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 11:33:49.062676 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 11:33:49.062813 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":55,"thread_start_us":398,"threads_started":1}
I20260924 11:33:49.064446 8622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:49.041563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54092 (local address 127.6.212.2:36373)
0924 11:33:49.054930 (+ 13367us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:49.054940 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:49.054966 (+ 26us) server_negotiation.cc:407] Connection header received
0924 11:33:49.055052 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:49.055060 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:49.055311 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:49.056262 (+ 951us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:33:49.056466 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:49.057920 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:49.060701 (+ 2781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:49.062100 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:49.063712 (+ 1612us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:49.063931 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:49.063998 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 11:33:49.064169 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13232,"thread_start_us":13161,"threads_started":1}
I20260924 11:33:49.066496 8620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:49.039960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35354 (local address 127.6.212.1:46675)
0924 11:33:49.051397 (+ 11437us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:49.051407 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:49.053976 (+ 2569us) server_negotiation.cc:407] Connection header received
0924 11:33:49.055061 (+ 1085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:49.055068 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:49.055339 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:49.055552 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:33:49.056845 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:49.058244 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:49.063657 (+ 5413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:49.065128 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:49.065231 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:49.065502 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:49.066051 (+ 549us) server_negotiation.cc:299] Negotiation successful
0924 11:33:49.066205 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11242,"thread_start_us":11118,"threads_started":1}
I20260924 11:33:49.066969 8618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:49.039097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.3:43581 (local address 127.0.0.1:53916)
0924 11:33:49.040399 (+ 1302us) negotiation.cc:107] Waiting for socket to connect
0924 11:33:49.040418 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:33:49.040540 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:33:49.050501 (+ 9961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:33:49.050509 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:33:49.050545 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:33:49.050954 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:49.050969 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:49.053582 (+ 2613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:49.053590 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:49.056748 (+ 3158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:49.056765 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:49.060834 (+ 4069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:49.060845 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:49.061086 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:49.061165 (+ 79us) client_negotiation.cc:769] Sending connection context
0924 11:33:49.061356 (+ 191us) client_negotiation.cc:240] Negotiation successful
0924 11:33:49.061571 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1161,"mutex_wait_us":5,"thread_start_us":1938,"threads_started":1}
I20260924 11:33:49.066975 8619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:49.039478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:46675 (local address 127.0.0.1:35354)
0924 11:33:49.053852 (+ 14374us) negotiation.cc:107] Waiting for socket to connect
0924 11:33:49.053874 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 11:33:49.054601 (+ 727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:33:49.055834 (+ 1233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:33:49.055841 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:33:49.055869 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:33:49.056650 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:49.056663 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:49.058622 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:49.058628 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:49.063467 (+ 4839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:49.063485 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:49.065731 (+ 2246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:49.065738 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:49.065873 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:49.065895 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 11:33:49.066009 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 11:33:49.066133 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14241,"mutex_wait_us":929,"thread_start_us":14091,"threads_started":1}
I20260924 11:33:49.069424 8395 tablet_service.cc:1467] Processing CreateTablet for tablet fa7f7043b2db4cf3a4d7c312c9fb7708 (DEFAULT_TABLE table=test-table [id=5fc749cba3c8441889260e106e3bf041]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:33:49.071084 8395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa7f7043b2db4cf3a4d7c312c9fb7708. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:33:49.071110 8256 tablet_service.cc:1467] Processing CreateTablet for tablet fa7f7043b2db4cf3a4d7c312c9fb7708 (DEFAULT_TABLE table=test-table [id=5fc749cba3c8441889260e106e3bf041]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:33:49.071111 8534 tablet_service.cc:1467] Processing CreateTablet for tablet fa7f7043b2db4cf3a4d7c312c9fb7708 (DEFAULT_TABLE table=test-table [id=5fc749cba3c8441889260e106e3bf041]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:33:49.072516 8256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa7f7043b2db4cf3a4d7c312c9fb7708. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:33:49.072614 8534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa7f7043b2db4cf3a4d7c312c9fb7708. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:33:49.093556 8623 tablet_bootstrap.cc:492] T fa7f7043b2db4cf3a4d7c312c9fb7708 P e7e9dd80d738487cb5e99a5d55eded82: Bootstrap starting.
I20260924 11:33:49.095595 8624 tablet_bootstrap.cc:492] T fa7f7043b2db4cf3a4d7c312c9fb7708 P 3aae29554c0f426299acdeed32fe630b: Bootstrap starting.
I20260924 11:33:49.096176 8625 tablet_bootstrap.cc:492] T fa7f7043b2db4cf3a4d7c312c9fb7708 P c48acea63acb4d6aaf797e3574a4cad3: Bootstrap starting.
I20260924 11:33:49.098701 8624 tablet_bootstrap.cc:654] T fa7f7043b2db4cf3a4d7c312c9fb7708 P 3aae29554c0f426299acdeed32fe630b: Neither blocks nor log segments found. Creating new log.
I20260924 11:33:49.098711 8625 tablet_bootstrap.cc:654] T fa7f7043b2db4cf3a4d7c312c9fb7708 P c48acea63acb4d6aaf797e3574a4cad3: Neither blocks nor log segments found. Creating new log.
I20260924 11:33:49.099946 8625 log.cc:824] T fa7f7043b2db4cf3a4d7c312c9fb7708 P c48acea63acb4d6aaf797e3574a4cad3: Log is configured to *not* fsync() on all Append() calls
I20260924 11:33:49.099996 8624 log.cc:824] T fa7f7043b2db4cf3a4d7c312c9fb7708 P 3aae29554c0f426299acdeed32fe630b: Log is configured to *not* fsync() on all Append() calls
I20260924 11:33:49.102665 8625 tablet_bootstrap.cc:492] T fa7f7043b2db4cf3a4d7c312c9fb7708 P c48acea63acb4d6aaf797e3574a4cad3: No bootstrap required, opened a new log
I20260924 11:33:49.102948 8625 ts_tablet_manager.cc:1397] T fa7f7043b2db4cf3a4d7c312c9fb7708 P c48acea63acb4d6aaf797e3574a4cad3: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 11:33:49.103206 8623 tablet_bootstrap.cc:654] T fa7f7043b2db4cf3a4d7c312c9fb7708 P e7e9dd80d738487cb5e99a5d55eded82: Neither blocks nor log segments found. Creating new log.
I20260924 11:33:49.104585 8623 log.cc:824] T fa7f7043b2db4cf3a4d7c312c9fb7708 P e7e9dd80d738487cb5e99a5d55eded82: Log is configured to *not* fsync() on all Append() calls
I20260924 11:33:49.105003 8624 tablet_bootstrap.cc:492] T fa7f7043b2db4cf3a4d7c312c9fb7708 P 3aae29554c0f426299acdeed32fe630b: No bootstrap required, opened a new log
I20260924 11:33:49.105320 8624 ts_tablet_manager.cc:1397] T fa7f7043b2db4cf3a4d7c312c9fb7708 P 3aae29554c0f426299acdeed32fe630b: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260924 11:33:49.107774 8623 tablet_bootstrap.cc:492] T fa7f7043b2db4cf3a4d7c312c9fb7708 P e7e9dd80d738487cb5e99a5d55eded82: No bootstrap required, opened a new log
I20260924 11:33:49.108121 8623 ts_tablet_manager.cc:1397] T fa7f7043b2db4cf3a4d7c312c9fb7708 P e7e9dd80d738487cb5e99a5d55eded82: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.000s
I20260924 11:33:49.111773 8625 raft_consensus.cc:348] T fa7f7043b2db4cf3a4d7c312c9fb7708 P c48acea63acb4d6aaf797e3574a4cad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aae29554c0f426299acdeed32fe630b" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36373 } } peers { permanent_uuid: "c48acea63acb4d6aaf797e3574a4cad3" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43581 } } peers { permanent_uuid: "e7e9dd80d738487cb5e99a5d55eded82" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46675 } }
I20260924 11:33:49.112171 8625 raft_consensus.cc:374] T fa7f7043b2db4cf3a4d7c312c9fb7708 P c48acea63acb4d6aaf797e3574a4cad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:33:49.112269 8625 raft_consensus.cc:729] T fa7f7043b2db4cf3a4d7c312c9fb7708 P c48acea63acb4d6aaf797e3574a4cad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c48acea63acb4d6aaf797e3574a4cad3, State: Initialized, Role: FOLLOWER
I20260924 11:33:49.112917 8625 consensus_queue.cc:260] T fa7f7043b2db4cf3a4d7c312c9fb7708 P c48acea63acb4d6aaf797e3574a4cad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aae29554c0f426299acdeed32fe630b" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36373 } } peers { permanent_uuid: "c48acea63acb4d6aaf797e3574a4cad3" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43581 } } peers { permanent_uuid: "e7e9dd80d738487cb5e99a5d55eded82" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46675 } }
I20260924 11:33:49.114022 8604 heartbeater.cc:503] Master 127.6.212.62:40625 was elected leader, sending a full tablet report...
I20260924 11:33:49.116248 8625 ts_tablet_manager.cc:1428] T fa7f7043b2db4cf3a4d7c312c9fb7708 P c48acea63acb4d6aaf797e3574a4cad3: Time spent starting tablet: real 0.013s user 0.007s sys 0.004s
I20260924 11:33:49.118551 8624 raft_consensus.cc:348] T fa7f7043b2db4cf3a4d7c312c9fb7708 P 3aae29554c0f426299acdeed32fe630b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aae29554c0f426299acdeed32fe630b" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36373 } } peers { permanent_uuid: "c48acea63acb4d6aaf797e3574a4cad3" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43581 } } peers { permanent_uuid: "e7e9dd80d738487cb5e99a5d55eded82" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46675 } }
I20260924 11:33:49.119060 8624 raft_consensus.cc:374] T fa7f7043b2db4cf3a4d7c312c9fb7708 P 3aae29554c0f426299acdeed32fe630b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:33:49.119182 8624 raft_consensus.cc:729] T fa7f7043b2db4cf3a4d7c312c9fb7708 P 3aae29554c0f426299acdeed32fe630b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aae29554c0f426299acdeed32fe630b, State: Initialized, Role: FOLLOWER
I20260924 11:33:49.119336 8623 raft_consensus.cc:348] T fa7f7043b2db4cf3a4d7c312c9fb7708 P e7e9dd80d738487cb5e99a5d55eded82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aae29554c0f426299acdeed32fe630b" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36373 } } peers { permanent_uuid: "c48acea63acb4d6aaf797e3574a4cad3" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43581 } } peers { permanent_uuid: "e7e9dd80d738487cb5e99a5d55eded82" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46675 } }
I20260924 11:33:49.119794 8623 raft_consensus.cc:374] T fa7f7043b2db4cf3a4d7c312c9fb7708 P e7e9dd80d738487cb5e99a5d55eded82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:33:49.119903 8623 raft_consensus.cc:729] T fa7f7043b2db4cf3a4d7c312c9fb7708 P e7e9dd80d738487cb5e99a5d55eded82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7e9dd80d738487cb5e99a5d55eded82, State: Initialized, Role: FOLLOWER
I20260924 11:33:49.119880 8624 consensus_queue.cc:260] T fa7f7043b2db4cf3a4d7c312c9fb7708 P 3aae29554c0f426299acdeed32fe630b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aae29554c0f426299acdeed32fe630b" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36373 } } peers { permanent_uuid: "c48acea63acb4d6aaf797e3574a4cad3" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43581 } } peers { permanent_uuid: "e7e9dd80d738487cb5e99a5d55eded82" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46675 } }
I20260924 11:33:49.120615 8623 consensus_queue.cc:260] T fa7f7043b2db4cf3a4d7c312c9fb7708 P e7e9dd80d738487cb5e99a5d55eded82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aae29554c0f426299acdeed32fe630b" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36373 } } peers { permanent_uuid: "c48acea63acb4d6aaf797e3574a4cad3" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43581 } } peers { permanent_uuid: "e7e9dd80d738487cb5e99a5d55eded82" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46675 } }
I20260924 11:33:49.120972 8465 heartbeater.cc:503] Master 127.6.212.62:40625 was elected leader, sending a full tablet report...
I20260924 11:33:49.122462 8624 ts_tablet_manager.cc:1428] T fa7f7043b2db4cf3a4d7c312c9fb7708 P 3aae29554c0f426299acdeed32fe630b: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
I20260924 11:33:49.123549 8623 ts_tablet_manager.cc:1428] T fa7f7043b2db4cf3a4d7c312c9fb7708 P e7e9dd80d738487cb5e99a5d55eded82: Time spent starting tablet: real 0.015s user 0.001s sys 0.014s
W20260924 11:33:49.188115 8327 tablet.cc:2369] T fa7f7043b2db4cf3a4d7c312c9fb7708 P e7e9dd80d738487cb5e99a5d55eded82: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:33:49.192451 8466 tablet.cc:2369] T fa7f7043b2db4cf3a4d7c312c9fb7708 P 3aae29554c0f426299acdeed32fe630b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:33:49.203054 8605 tablet.cc:2369] T fa7f7043b2db4cf3a4d7c312c9fb7708 P c48acea63acb4d6aaf797e3574a4cad3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:33:49.220409 8629 raft_consensus.cc:482] T fa7f7043b2db4cf3a4d7c312c9fb7708 P c48acea63acb4d6aaf797e3574a4cad3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:33:49.220685 8629 raft_consensus.cc:504] T fa7f7043b2db4cf3a4d7c312c9fb7708 P c48acea63acb4d6aaf797e3574a4cad3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aae29554c0f426299acdeed32fe630b" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36373 } } peers { permanent_uuid: "c48acea63acb4d6aaf797e3574a4cad3" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43581 } } peers { permanent_uuid: "e7e9dd80d738487cb5e99a5d55eded82" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46675 } }
I20260924 11:33:49.222373 8629 leader_election.cc:288] T fa7f7043b2db4cf3a4d7c312c9fb7708 P c48acea63acb4d6aaf797e3574a4cad3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aae29554c0f426299acdeed32fe630b (127.6.212.2:36373), e7e9dd80d738487cb5e99a5d55eded82 (127.6.212.1:46675)
I20260924 11:33:49.229239 8620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:49.222847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:60799 (local address 127.6.212.1:46675)
0924 11:33:49.223075 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:49.223081 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:49.223244 (+ 163us) server_negotiation.cc:407] Connection header received
0924 11:33:49.223492 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:49.223497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:49.223589 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:49.223708 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:33:49.224250 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:49.225249 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:49.227158 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:49.228396 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:49.228510 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:49.228592 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:49.228967 (+ 375us) server_negotiation.cc:299] Negotiation successful
0924 11:33:49.229064 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260924 11:33:49.229323 8493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:49.222633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:46675 (local address 127.6.212.3:60799)
0924 11:33:49.223100 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0924 11:33:49.223128 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 11:33:49.223301 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:33:49.223767 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:33:49.223774 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:33:49.223804 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:33:49.224069 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:49.224080 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:49.225376 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:49.225382 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:49.226989 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:49.227000 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:49.228555 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:49.228561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:49.228760 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:49.228794 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 11:33:49.228894 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 11:33:49.229034 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":8}
I20260924 11:33:49.230268 8622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:49.222877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:44817 (local address 127.6.212.2:36373)
0924 11:33:49.223062 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:49.223069 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:49.223745 (+ 676us) server_negotiation.cc:407] Connection header received
0924 11:33:49.223898 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:49.223904 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:49.223973 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:49.224072 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:33:49.224743 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:49.225582 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:49.228219 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:49.229267 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:49.229395 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:49.229532 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:49.230010 (+ 478us) server_negotiation.cc:299] Negotiation successful
0924 11:33:49.230099 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 11:33:49.230846 8632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:49.222633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.2:36373 (local address 127.6.212.3:44817)
0924 11:33:49.223635 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0924 11:33:49.223654 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:33:49.223767 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:33:49.224095 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:33:49.224099 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:33:49.224117 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:33:49.224587 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:49.224601 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:49.225752 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:49.225755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:49.228056 (+ 2301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:49.228073 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:49.229660 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:49.229664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:49.229824 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:49.229856 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 11:33:49.229957 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 11:33:49.230083 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":887,"mutex_wait_us":157,"thread_start_us":538,"threads_started":1}
I20260924 11:33:49.230311 8280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7f7043b2db4cf3a4d7c312c9fb7708" candidate_uuid: "c48acea63acb4d6aaf797e3574a4cad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7e9dd80d738487cb5e99a5d55eded82" is_pre_election: true
I20260924 11:33:49.231272 8280 raft_consensus.cc:2454] T fa7f7043b2db4cf3a4d7c312c9fb7708 P e7e9dd80d738487cb5e99a5d55eded82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c48acea63acb4d6aaf797e3574a4cad3 in term 0.
I20260924 11:33:49.231870 8420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7f7043b2db4cf3a4d7c312c9fb7708" candidate_uuid: "c48acea63acb4d6aaf797e3574a4cad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aae29554c0f426299acdeed32fe630b" is_pre_election: true
I20260924 11:33:49.231982 8487 leader_election.cc:302] T fa7f7043b2db4cf3a4d7c312c9fb7708 P c48acea63acb4d6aaf797e3574a4cad3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c48acea63acb4d6aaf797e3574a4cad3, e7e9dd80d738487cb5e99a5d55eded82; no voters:
I20260924 11:33:49.232360 8629 raft_consensus.cc:2781] T fa7f7043b2db4cf3a4d7c312c9fb7708 P c48acea63acb4d6aaf797e3574a4cad3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:33:49.232336 8420 raft_consensus.cc:2454] T fa7f7043b2db4cf3a4d7c312c9fb7708 P 3aae29554c0f426299acdeed32fe630b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c48acea63acb4d6aaf797e3574a4cad3 in term 0.
I20260924 11:33:49.232509 8629 raft_consensus.cc:482] T fa7f7043b2db4cf3a4d7c312c9fb7708 P c48acea63acb4d6aaf797e3574a4cad3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:33:49.232610 8629 raft_consensus.cc:3037] T fa7f7043b2db4cf3a4d7c312c9fb7708 P c48acea63acb4d6aaf797e3574a4cad3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:33:49.234952 8629 raft_consensus.cc:504] T fa7f7043b2db4cf3a4d7c312c9fb7708 P c48acea63acb4d6aaf797e3574a4cad3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aae29554c0f426299acdeed32fe630b" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36373 } } peers { permanent_uuid: "c48acea63acb4d6aaf797e3574a4cad3" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43581 } } peers { permanent_uuid: "e7e9dd80d738487cb5e99a5d55eded82" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46675 } }
I20260924 11:33:49.235577 8629 leader_election.cc:288] T fa7f7043b2db4cf3a4d7c312c9fb7708 P c48acea63acb4d6aaf797e3574a4cad3 [CANDIDATE]: Term 1 election: Requested vote from peers 3aae29554c0f426299acdeed32fe630b (127.6.212.2:36373), e7e9dd80d738487cb5e99a5d55eded82 (127.6.212.1:46675)
I20260924 11:33:49.236013 8280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7f7043b2db4cf3a4d7c312c9fb7708" candidate_uuid: "c48acea63acb4d6aaf797e3574a4cad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7e9dd80d738487cb5e99a5d55eded82"
I20260924 11:33:49.236147 8420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7f7043b2db4cf3a4d7c312c9fb7708" candidate_uuid: "c48acea63acb4d6aaf797e3574a4cad3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aae29554c0f426299acdeed32fe630b"
I20260924 11:33:49.236193 8280 raft_consensus.cc:3037] T fa7f7043b2db4cf3a4d7c312c9fb7708 P e7e9dd80d738487cb5e99a5d55eded82 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:33:49.236349 8420 raft_consensus.cc:3037] T fa7f7043b2db4cf3a4d7c312c9fb7708 P 3aae29554c0f426299acdeed32fe630b [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:33:49.238511 8280 raft_consensus.cc:2454] T fa7f7043b2db4cf3a4d7c312c9fb7708 P e7e9dd80d738487cb5e99a5d55eded82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c48acea63acb4d6aaf797e3574a4cad3 in term 1.
I20260924 11:33:49.238519 8420 raft_consensus.cc:2454] T fa7f7043b2db4cf3a4d7c312c9fb7708 P 3aae29554c0f426299acdeed32fe630b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c48acea63acb4d6aaf797e3574a4cad3 in term 1.
I20260924 11:33:49.238878 8487 leader_election.cc:302] T fa7f7043b2db4cf3a4d7c312c9fb7708 P c48acea63acb4d6aaf797e3574a4cad3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c48acea63acb4d6aaf797e3574a4cad3, e7e9dd80d738487cb5e99a5d55eded82; no voters:
I20260924 11:33:49.239152 8629 raft_consensus.cc:2781] T fa7f7043b2db4cf3a4d7c312c9fb7708 P c48acea63acb4d6aaf797e3574a4cad3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:33:49.239598 8629 raft_consensus.cc:686] T fa7f7043b2db4cf3a4d7c312c9fb7708 P c48acea63acb4d6aaf797e3574a4cad3 [term 1 LEADER]: Becoming Leader. State: Replica: c48acea63acb4d6aaf797e3574a4cad3, State: Running, Role: LEADER
I20260924 11:33:49.240056 8629 consensus_queue.cc:237] T fa7f7043b2db4cf3a4d7c312c9fb7708 P c48acea63acb4d6aaf797e3574a4cad3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aae29554c0f426299acdeed32fe630b" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36373 } } peers { permanent_uuid: "c48acea63acb4d6aaf797e3574a4cad3" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43581 } } peers { permanent_uuid: "e7e9dd80d738487cb5e99a5d55eded82" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46675 } }
I20260924 11:33:49.245162 8119 catalog_manager.cc:5495] T fa7f7043b2db4cf3a4d7c312c9fb7708 P c48acea63acb4d6aaf797e3574a4cad3 reported cstate change: term changed from 0 to 1, leader changed from <none> to c48acea63acb4d6aaf797e3574a4cad3 (127.6.212.3). New cstate: current_term: 1 leader_uuid: "c48acea63acb4d6aaf797e3574a4cad3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aae29554c0f426299acdeed32fe630b" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c48acea63acb4d6aaf797e3574a4cad3" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7e9dd80d738487cb5e99a5d55eded82" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46675 } health_report { overall_health: UNKNOWN } } }
I20260924 11:33:49.306565 8621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:49.300650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53930 (local address 127.6.212.3:43581)
0924 11:33:49.300838 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:49.300843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:49.300945 (+ 102us) server_negotiation.cc:407] Connection header received
0924 11:33:49.301169 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:49.301172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:49.301245 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:49.301363 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:33:49.301785 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:49.302499 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:49.304105 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:49.304494 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:49.304620 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:49.305347 (+ 727us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:33:49.306012 (+ 665us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:33:49.306203 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:49.306264 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 11:33:49.306324 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 11:33:49.315152 8280 raft_consensus.cc:1261] T fa7f7043b2db4cf3a4d7c312c9fb7708 P e7e9dd80d738487cb5e99a5d55eded82 [term 1 FOLLOWER]: Refusing update from remote peer c48acea63acb4d6aaf797e3574a4cad3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:33:49.315269 8420 raft_consensus.cc:1261] T fa7f7043b2db4cf3a4d7c312c9fb7708 P 3aae29554c0f426299acdeed32fe630b [term 1 FOLLOWER]: Refusing update from remote peer c48acea63acb4d6aaf797e3574a4cad3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:33:49.316236 8633 consensus_queue.cc:1035] T fa7f7043b2db4cf3a4d7c312c9fb7708 P c48acea63acb4d6aaf797e3574a4cad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aae29554c0f426299acdeed32fe630b" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:33:49.316602 8629 consensus_queue.cc:1035] T fa7f7043b2db4cf3a4d7c312c9fb7708 P c48acea63acb4d6aaf797e3574a4cad3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7e9dd80d738487cb5e99a5d55eded82" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:33:49.327077 8636 mvcc.cc:205] Tried to move back new op lower bound from 7332862481665212416 to 7332862481370963968. Current Snapshot: MvccSnapshot[applied={T|T < 7332862481665212416}]
I20260924 11:33:49.334218 8637 mvcc.cc:205] Tried to move back new op lower bound from 7332862481665212416 to 7332862481370963968. Current Snapshot: MvccSnapshot[applied={T|T < 7332862481665212416}]
I20260924 11:33:49.343911 8638 mvcc.cc:205] Tried to move back new op lower bound from 7332862481665212416 to 7332862481370963968. Current Snapshot: MvccSnapshot[applied={T|T < 7332862481665212416}]
I20260924 11:33:49.352182 8119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:33:49.352424 8119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:33:49.356020 8119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 092352e3835147d59f8e8772a70a9475: Sending DeleteTablet for 3 replicas of tablet fa7f7043b2db4cf3a4d7c312c9fb7708
I20260924 11:33:49.357151 8395 tablet_service.cc:1514] Processing DeleteTablet for tablet fa7f7043b2db4cf3a4d7c312c9fb7708 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:33:49 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:54092
I20260924 11:33:49.357158 8534 tablet_service.cc:1514] Processing DeleteTablet for tablet fa7f7043b2db4cf3a4d7c312c9fb7708 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:33:49 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:53916
I20260924 11:33:49.357515 8256 tablet_service.cc:1514] Processing DeleteTablet for tablet fa7f7043b2db4cf3a4d7c312c9fb7708 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:33:49 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:35354
I20260924 11:33:49.359726 8119 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52798:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:33:49.361147 8119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:33:49.368199 8649 tablet_replica.cc:330] T fa7f7043b2db4cf3a4d7c312c9fb7708 P e7e9dd80d738487cb5e99a5d55eded82: stopping tablet replica
I20260924 11:33:49.368662 8649 raft_consensus.cc:2229] T fa7f7043b2db4cf3a4d7c312c9fb7708 P e7e9dd80d738487cb5e99a5d55eded82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:33:49.369199 8649 raft_consensus.cc:2258] T fa7f7043b2db4cf3a4d7c312c9fb7708 P e7e9dd80d738487cb5e99a5d55eded82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:33:49.370463 8648 tablet_replica.cc:330] T fa7f7043b2db4cf3a4d7c312c9fb7708 P 3aae29554c0f426299acdeed32fe630b: stopping tablet replica
I20260924 11:33:49.371387 8647 tablet_replica.cc:330] T fa7f7043b2db4cf3a4d7c312c9fb7708 P c48acea63acb4d6aaf797e3574a4cad3: stopping tablet replica
I20260924 11:33:49.371820 8647 raft_consensus.cc:2229] T fa7f7043b2db4cf3a4d7c312c9fb7708 P c48acea63acb4d6aaf797e3574a4cad3 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:33:49.372749 8256 tablet_service.cc:1467] Processing CreateTablet for tablet 9501ff7764b4445fb430ff283a8df0b1 (DEFAULT_TABLE table=test-table [id=07cdb4a3f73a4d22b0e4fc9584b491f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:33:49.373181 8256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9501ff7764b4445fb430ff283a8df0b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:33:49.373263 8534 tablet_service.cc:1467] Processing CreateTablet for tablet 9501ff7764b4445fb430ff283a8df0b1 (DEFAULT_TABLE table=test-table [id=07cdb4a3f73a4d22b0e4fc9584b491f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:33:49.373740 8534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9501ff7764b4445fb430ff283a8df0b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:33:49.370862 8648 raft_consensus.cc:2229] T fa7f7043b2db4cf3a4d7c312c9fb7708 P 3aae29554c0f426299acdeed32fe630b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:33:49.372929 8395 tablet_service.cc:1467] Processing CreateTablet for tablet 9501ff7764b4445fb430ff283a8df0b1 (DEFAULT_TABLE table=test-table [id=07cdb4a3f73a4d22b0e4fc9584b491f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:33:49.375759 8395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9501ff7764b4445fb430ff283a8df0b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:33:49.375964 8648 raft_consensus.cc:2258] T fa7f7043b2db4cf3a4d7c312c9fb7708 P 3aae29554c0f426299acdeed32fe630b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:33:49.376396 8647 raft_consensus.cc:2258] T fa7f7043b2db4cf3a4d7c312c9fb7708 P c48acea63acb4d6aaf797e3574a4cad3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:33:49.378692 8649 ts_tablet_manager.cc:1905] T fa7f7043b2db4cf3a4d7c312c9fb7708 P e7e9dd80d738487cb5e99a5d55eded82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:33:49.380263 8623 tablet_bootstrap.cc:492] T 9501ff7764b4445fb430ff283a8df0b1 P e7e9dd80d738487cb5e99a5d55eded82: Bootstrap starting.
I20260924 11:33:49.381726 8623 tablet_bootstrap.cc:654] T 9501ff7764b4445fb430ff283a8df0b1 P e7e9dd80d738487cb5e99a5d55eded82: Neither blocks nor log segments found. Creating new log.
I20260924 11:33:49.382530 8624 tablet_bootstrap.cc:492] T 9501ff7764b4445fb430ff283a8df0b1 P 3aae29554c0f426299acdeed32fe630b: Bootstrap starting.
I20260924 11:33:49.385622 8647 ts_tablet_manager.cc:1905] T fa7f7043b2db4cf3a4d7c312c9fb7708 P c48acea63acb4d6aaf797e3574a4cad3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:33:49.386639 8649 ts_tablet_manager.cc:1918] T fa7f7043b2db4cf3a4d7c312c9fb7708 P e7e9dd80d738487cb5e99a5d55eded82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:33:49.386776 8649 log.cc:1196] T fa7f7043b2db4cf3a4d7c312c9fb7708 P e7e9dd80d738487cb5e99a5d55eded82: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-0/wal/wals/fa7f7043b2db4cf3a4d7c312c9fb7708
I20260924 11:33:49.387286 8649 ts_tablet_manager.cc:1939] T fa7f7043b2db4cf3a4d7c312c9fb7708 P e7e9dd80d738487cb5e99a5d55eded82: Deleting consensus metadata
I20260924 11:33:49.390157 8648 ts_tablet_manager.cc:1905] T fa7f7043b2db4cf3a4d7c312c9fb7708 P 3aae29554c0f426299acdeed32fe630b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:33:49.390618 8105 catalog_manager.cc:4841] TS e7e9dd80d738487cb5e99a5d55eded82 (127.6.212.1:46675): tablet fa7f7043b2db4cf3a4d7c312c9fb7708 (table test-table [id=5fc749cba3c8441889260e106e3bf041]) successfully deleted
I20260924 11:33:49.390187 8624 tablet_bootstrap.cc:654] T 9501ff7764b4445fb430ff283a8df0b1 P 3aae29554c0f426299acdeed32fe630b: Neither blocks nor log segments found. Creating new log.
I20260924 11:33:49.392036 8623 tablet_bootstrap.cc:492] T 9501ff7764b4445fb430ff283a8df0b1 P e7e9dd80d738487cb5e99a5d55eded82: No bootstrap required, opened a new log
I20260924 11:33:49.392231 8623 ts_tablet_manager.cc:1397] T 9501ff7764b4445fb430ff283a8df0b1 P e7e9dd80d738487cb5e99a5d55eded82: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20260924 11:33:49.393131 8623 raft_consensus.cc:348] T 9501ff7764b4445fb430ff283a8df0b1 P e7e9dd80d738487cb5e99a5d55eded82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e9dd80d738487cb5e99a5d55eded82" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46675 } } peers { permanent_uuid: "3aae29554c0f426299acdeed32fe630b" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36373 } } peers { permanent_uuid: "c48acea63acb4d6aaf797e3574a4cad3" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43581 } }
I20260924 11:33:49.393530 8623 raft_consensus.cc:374] T 9501ff7764b4445fb430ff283a8df0b1 P e7e9dd80d738487cb5e99a5d55eded82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:33:49.393688 8623 raft_consensus.cc:729] T 9501ff7764b4445fb430ff283a8df0b1 P e7e9dd80d738487cb5e99a5d55eded82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7e9dd80d738487cb5e99a5d55eded82, State: Initialized, Role: FOLLOWER
I20260924 11:33:49.394069 8623 consensus_queue.cc:260] T 9501ff7764b4445fb430ff283a8df0b1 P e7e9dd80d738487cb5e99a5d55eded82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e9dd80d738487cb5e99a5d55eded82" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46675 } } peers { permanent_uuid: "3aae29554c0f426299acdeed32fe630b" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36373 } } peers { permanent_uuid: "c48acea63acb4d6aaf797e3574a4cad3" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43581 } }
I20260924 11:33:49.394538 8647 ts_tablet_manager.cc:1918] T fa7f7043b2db4cf3a4d7c312c9fb7708 P c48acea63acb4d6aaf797e3574a4cad3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:33:49.394685 8647 log.cc:1196] T fa7f7043b2db4cf3a4d7c312c9fb7708 P c48acea63acb4d6aaf797e3574a4cad3: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-2/wal/wals/fa7f7043b2db4cf3a4d7c312c9fb7708
I20260924 11:33:49.395303 8647 ts_tablet_manager.cc:1939] T fa7f7043b2db4cf3a4d7c312c9fb7708 P c48acea63acb4d6aaf797e3574a4cad3: Deleting consensus metadata
I20260924 11:33:49.395969 8623 ts_tablet_manager.cc:1428] T 9501ff7764b4445fb430ff283a8df0b1 P e7e9dd80d738487cb5e99a5d55eded82: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260924 11:33:49.397353 8648 ts_tablet_manager.cc:1918] T fa7f7043b2db4cf3a4d7c312c9fb7708 P 3aae29554c0f426299acdeed32fe630b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:33:49.397490 8104 catalog_manager.cc:4841] TS c48acea63acb4d6aaf797e3574a4cad3 (127.6.212.3:43581): tablet fa7f7043b2db4cf3a4d7c312c9fb7708 (table test-table [id=5fc749cba3c8441889260e106e3bf041]) successfully deleted
I20260924 11:33:49.397572 8648 log.cc:1196] T fa7f7043b2db4cf3a4d7c312c9fb7708 P 3aae29554c0f426299acdeed32fe630b: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-1/wal/wals/fa7f7043b2db4cf3a4d7c312c9fb7708
I20260924 11:33:49.398206 8648 ts_tablet_manager.cc:1939] T fa7f7043b2db4cf3a4d7c312c9fb7708 P 3aae29554c0f426299acdeed32fe630b: Deleting consensus metadata
I20260924 11:33:49.400717 8106 catalog_manager.cc:4841] TS 3aae29554c0f426299acdeed32fe630b (127.6.212.2:36373): tablet fa7f7043b2db4cf3a4d7c312c9fb7708 (table test-table [id=5fc749cba3c8441889260e106e3bf041]) successfully deleted
I20260924 11:33:49.401165 8625 tablet_bootstrap.cc:492] T 9501ff7764b4445fb430ff283a8df0b1 P c48acea63acb4d6aaf797e3574a4cad3: Bootstrap starting.
I20260924 11:33:49.403007 8625 tablet_bootstrap.cc:654] T 9501ff7764b4445fb430ff283a8df0b1 P c48acea63acb4d6aaf797e3574a4cad3: Neither blocks nor log segments found. Creating new log.
I20260924 11:33:49.403154 8624 tablet_bootstrap.cc:492] T 9501ff7764b4445fb430ff283a8df0b1 P 3aae29554c0f426299acdeed32fe630b: No bootstrap required, opened a new log
I20260924 11:33:49.403254 8624 ts_tablet_manager.cc:1397] T 9501ff7764b4445fb430ff283a8df0b1 P 3aae29554c0f426299acdeed32fe630b: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.004s
I20260924 11:33:49.403925 8624 raft_consensus.cc:348] T 9501ff7764b4445fb430ff283a8df0b1 P 3aae29554c0f426299acdeed32fe630b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e9dd80d738487cb5e99a5d55eded82" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46675 } } peers { permanent_uuid: "3aae29554c0f426299acdeed32fe630b" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36373 } } peers { permanent_uuid: "c48acea63acb4d6aaf797e3574a4cad3" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43581 } }
I20260924 11:33:49.404197 8624 raft_consensus.cc:374] T 9501ff7764b4445fb430ff283a8df0b1 P 3aae29554c0f426299acdeed32fe630b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:33:49.404297 8624 raft_consensus.cc:729] T 9501ff7764b4445fb430ff283a8df0b1 P 3aae29554c0f426299acdeed32fe630b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aae29554c0f426299acdeed32fe630b, State: Initialized, Role: FOLLOWER
I20260924 11:33:49.404561 8624 consensus_queue.cc:260] T 9501ff7764b4445fb430ff283a8df0b1 P 3aae29554c0f426299acdeed32fe630b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e9dd80d738487cb5e99a5d55eded82" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46675 } } peers { permanent_uuid: "3aae29554c0f426299acdeed32fe630b" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36373 } } peers { permanent_uuid: "c48acea63acb4d6aaf797e3574a4cad3" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43581 } }
I20260924 11:33:49.405227 8624 ts_tablet_manager.cc:1428] T 9501ff7764b4445fb430ff283a8df0b1 P 3aae29554c0f426299acdeed32fe630b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 11:33:49.405555 8625 tablet_bootstrap.cc:492] T 9501ff7764b4445fb430ff283a8df0b1 P c48acea63acb4d6aaf797e3574a4cad3: No bootstrap required, opened a new log
I20260924 11:33:49.405692 8625 ts_tablet_manager.cc:1397] T 9501ff7764b4445fb430ff283a8df0b1 P c48acea63acb4d6aaf797e3574a4cad3: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260924 11:33:49.406528 8625 raft_consensus.cc:348] T 9501ff7764b4445fb430ff283a8df0b1 P c48acea63acb4d6aaf797e3574a4cad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e9dd80d738487cb5e99a5d55eded82" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46675 } } peers { permanent_uuid: "3aae29554c0f426299acdeed32fe630b" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36373 } } peers { permanent_uuid: "c48acea63acb4d6aaf797e3574a4cad3" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43581 } }
I20260924 11:33:49.406787 8625 raft_consensus.cc:374] T 9501ff7764b4445fb430ff283a8df0b1 P c48acea63acb4d6aaf797e3574a4cad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:33:49.406898 8625 raft_consensus.cc:729] T 9501ff7764b4445fb430ff283a8df0b1 P c48acea63acb4d6aaf797e3574a4cad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c48acea63acb4d6aaf797e3574a4cad3, State: Initialized, Role: FOLLOWER
I20260924 11:33:49.407155 8625 consensus_queue.cc:260] T 9501ff7764b4445fb430ff283a8df0b1 P c48acea63acb4d6aaf797e3574a4cad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e9dd80d738487cb5e99a5d55eded82" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46675 } } peers { permanent_uuid: "3aae29554c0f426299acdeed32fe630b" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36373 } } peers { permanent_uuid: "c48acea63acb4d6aaf797e3574a4cad3" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43581 } }
I20260924 11:33:49.408288 8625 ts_tablet_manager.cc:1428] T 9501ff7764b4445fb430ff283a8df0b1 P c48acea63acb4d6aaf797e3574a4cad3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 11:33:49.547737 8630 raft_consensus.cc:482] T 9501ff7764b4445fb430ff283a8df0b1 P 3aae29554c0f426299acdeed32fe630b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:33:49.548048 8630 raft_consensus.cc:504] T 9501ff7764b4445fb430ff283a8df0b1 P 3aae29554c0f426299acdeed32fe630b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e9dd80d738487cb5e99a5d55eded82" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46675 } } peers { permanent_uuid: "3aae29554c0f426299acdeed32fe630b" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36373 } } peers { permanent_uuid: "c48acea63acb4d6aaf797e3574a4cad3" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43581 } }
I20260924 11:33:49.549675 8630 leader_election.cc:288] T 9501ff7764b4445fb430ff283a8df0b1 P 3aae29554c0f426299acdeed32fe630b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7e9dd80d738487cb5e99a5d55eded82 (127.6.212.1:46675), c48acea63acb4d6aaf797e3574a4cad3 (127.6.212.3:43581)
I20260924 11:33:49.557984 8620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:49.550844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:53913 (local address 127.6.212.1:46675)
0924 11:33:49.551022 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:49.551027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:49.551045 (+ 18us) server_negotiation.cc:407] Connection header received
0924 11:33:49.551129 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:49.551135 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:49.551213 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:49.551315 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:33:49.552445 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:49.553542 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:49.555974 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:49.556911 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:49.557094 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:49.557215 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:49.557571 (+ 356us) server_negotiation.cc:299] Negotiation successful
0924 11:33:49.557706 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 11:33:49.558245 8621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:49.550488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:40503 (local address 127.6.212.3:43581)
0924 11:33:49.550670 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:49.550678 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:49.550882 (+ 204us) server_negotiation.cc:407] Connection header received
0924 11:33:49.551129 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:49.551141 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:49.551203 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:49.551251 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:33:49.552546 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:49.553297 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:49.555918 (+ 2621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:49.557316 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:49.557437 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:49.557521 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:49.557844 (+ 323us) server_negotiation.cc:299] Negotiation successful
0924 11:33:49.557971 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 11:33:49.558683 8654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:49.549989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.3:43581 (local address 127.6.212.2:40503)
0924 11:33:49.550789 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0924 11:33:49.550808 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:33:49.550972 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:33:49.551510 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:33:49.551519 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:33:49.551559 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:33:49.552059 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:49.552069 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:49.553440 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:49.553445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:49.555737 (+ 2292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:49.555752 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:49.557526 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:49.557535 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:49.557684 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:49.557713 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 11:33:49.557799 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 11:33:49.557922 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"mutex_wait_us":43,"spinlock_wait_cycles":896,"thread_start_us":945,"threads_started":1}
I20260924 11:33:49.558754 8653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:49.549884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:46675 (local address 127.6.212.2:53913)
0924 11:33:49.550643 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0924 11:33:49.550673 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 11:33:49.550837 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:33:49.551382 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:33:49.551387 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:33:49.551416 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:33:49.551854 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:49.551873 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:49.553665 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:49.553671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:49.555795 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:49.555807 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:49.557190 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:49.557199 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:49.557350 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:49.557395 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 11:33:49.558227 (+ 832us) client_negotiation.cc:240] Negotiation successful
0924 11:33:49.558369 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":7,"spinlock_wait_cycles":29312,"thread_start_us":311,"threads_started":1}
I20260924 11:33:49.559329 8280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9501ff7764b4445fb430ff283a8df0b1" candidate_uuid: "3aae29554c0f426299acdeed32fe630b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7e9dd80d738487cb5e99a5d55eded82" is_pre_election: true
I20260924 11:33:49.559609 8280 raft_consensus.cc:2454] T 9501ff7764b4445fb430ff283a8df0b1 P e7e9dd80d738487cb5e99a5d55eded82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aae29554c0f426299acdeed32fe630b in term 0.
I20260924 11:33:49.559721 8554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9501ff7764b4445fb430ff283a8df0b1" candidate_uuid: "3aae29554c0f426299acdeed32fe630b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c48acea63acb4d6aaf797e3574a4cad3" is_pre_election: true
I20260924 11:33:49.560144 8554 raft_consensus.cc:2454] T 9501ff7764b4445fb430ff283a8df0b1 P c48acea63acb4d6aaf797e3574a4cad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aae29554c0f426299acdeed32fe630b in term 0.
I20260924 11:33:49.560101 8348 leader_election.cc:302] T 9501ff7764b4445fb430ff283a8df0b1 P 3aae29554c0f426299acdeed32fe630b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aae29554c0f426299acdeed32fe630b, e7e9dd80d738487cb5e99a5d55eded82; no voters:
I20260924 11:33:49.560509 8630 raft_consensus.cc:2781] T 9501ff7764b4445fb430ff283a8df0b1 P 3aae29554c0f426299acdeed32fe630b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:33:49.560636 8630 raft_consensus.cc:482] T 9501ff7764b4445fb430ff283a8df0b1 P 3aae29554c0f426299acdeed32fe630b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:33:49.560714 8630 raft_consensus.cc:3037] T 9501ff7764b4445fb430ff283a8df0b1 P 3aae29554c0f426299acdeed32fe630b [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:33:49.563210 8630 raft_consensus.cc:504] T 9501ff7764b4445fb430ff283a8df0b1 P 3aae29554c0f426299acdeed32fe630b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e9dd80d738487cb5e99a5d55eded82" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46675 } } peers { permanent_uuid: "3aae29554c0f426299acdeed32fe630b" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36373 } } peers { permanent_uuid: "c48acea63acb4d6aaf797e3574a4cad3" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43581 } }
I20260924 11:33:49.563942 8630 leader_election.cc:288] T 9501ff7764b4445fb430ff283a8df0b1 P 3aae29554c0f426299acdeed32fe630b [CANDIDATE]: Term 1 election: Requested vote from peers e7e9dd80d738487cb5e99a5d55eded82 (127.6.212.1:46675), c48acea63acb4d6aaf797e3574a4cad3 (127.6.212.3:43581)
I20260924 11:33:49.564299 8280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9501ff7764b4445fb430ff283a8df0b1" candidate_uuid: "3aae29554c0f426299acdeed32fe630b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7e9dd80d738487cb5e99a5d55eded82"
I20260924 11:33:49.564520 8280 raft_consensus.cc:3037] T 9501ff7764b4445fb430ff283a8df0b1 P e7e9dd80d738487cb5e99a5d55eded82 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:33:49.564740 8554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9501ff7764b4445fb430ff283a8df0b1" candidate_uuid: "3aae29554c0f426299acdeed32fe630b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c48acea63acb4d6aaf797e3574a4cad3"
I20260924 11:33:49.564893 8554 raft_consensus.cc:3037] T 9501ff7764b4445fb430ff283a8df0b1 P c48acea63acb4d6aaf797e3574a4cad3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:33:49.567492 8280 raft_consensus.cc:2454] T 9501ff7764b4445fb430ff283a8df0b1 P e7e9dd80d738487cb5e99a5d55eded82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aae29554c0f426299acdeed32fe630b in term 1.
I20260924 11:33:49.567530 8554 raft_consensus.cc:2454] T 9501ff7764b4445fb430ff283a8df0b1 P c48acea63acb4d6aaf797e3574a4cad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aae29554c0f426299acdeed32fe630b in term 1.
I20260924 11:33:49.567883 8348 leader_election.cc:302] T 9501ff7764b4445fb430ff283a8df0b1 P 3aae29554c0f426299acdeed32fe630b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aae29554c0f426299acdeed32fe630b, e7e9dd80d738487cb5e99a5d55eded82; no voters:
I20260924 11:33:49.568207 8630 raft_consensus.cc:2781] T 9501ff7764b4445fb430ff283a8df0b1 P 3aae29554c0f426299acdeed32fe630b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:33:49.568765 8630 raft_consensus.cc:686] T 9501ff7764b4445fb430ff283a8df0b1 P 3aae29554c0f426299acdeed32fe630b [term 1 LEADER]: Becoming Leader. State: Replica: 3aae29554c0f426299acdeed32fe630b, State: Running, Role: LEADER
I20260924 11:33:49.569180 8630 consensus_queue.cc:237] T 9501ff7764b4445fb430ff283a8df0b1 P 3aae29554c0f426299acdeed32fe630b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e9dd80d738487cb5e99a5d55eded82" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46675 } } peers { permanent_uuid: "3aae29554c0f426299acdeed32fe630b" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36373 } } peers { permanent_uuid: "c48acea63acb4d6aaf797e3574a4cad3" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43581 } }
I20260924 11:33:49.573338 8118 catalog_manager.cc:5495] T 9501ff7764b4445fb430ff283a8df0b1 P 3aae29554c0f426299acdeed32fe630b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aae29554c0f426299acdeed32fe630b (127.6.212.2). New cstate: current_term: 1 leader_uuid: "3aae29554c0f426299acdeed32fe630b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e9dd80d738487cb5e99a5d55eded82" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aae29554c0f426299acdeed32fe630b" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c48acea63acb4d6aaf797e3574a4cad3" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43581 } health_report { overall_health: UNKNOWN } } }
Sep 24 11:33:49 dist-test-slave-cq7c krb5kdc[8074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249627, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:33:49.665138 8662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:49.649977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:40625 (local address 127.0.0.1:52800)
0924 11:33:49.650695 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0924 11:33:49.650722 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 11:33:49.650900 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:33:49.651340 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:33:49.651344 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:33:49.652523 (+ 1179us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:33:49.652731 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:49.652740 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:49.653805 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:49.653813 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:49.654721 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:49.654731 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:49.655551 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:49.655556 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:49.655749 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:49.656531 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:33:49.656553 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:33:49.658932 (+ 2379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:33:49.662501 (+ 3569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:33:49.662515 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:33:49.662524 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:33:49.662921 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:33:49.663283 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:33:49.663288 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:33:49.663291 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:33:49.663481 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:33:49.664010 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:33:49.664018 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:33:49.664176 (+ 158us) client_negotiation.cc:769] Sending connection context
0924 11:33:49.664337 (+ 161us) client_negotiation.cc:240] Negotiation successful
0924 11:33:49.664663 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"mutex_wait_us":26,"thread_start_us":363,"threads_started":1}
I20260924 11:33:49.665156 8663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:49.650134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52800 (local address 127.6.212.62:40625)
0924 11:33:49.651086 (+ 952us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:49.651090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:49.651108 (+ 18us) server_negotiation.cc:407] Connection header received
0924 11:33:49.651148 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:49.651151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:49.651201 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:49.651317 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:33:49.652894 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:49.653676 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:49.654856 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:49.655340 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:49.655532 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:49.659132 (+ 3600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:33:49.659138 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:33:49.659143 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:33:49.659179 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:33:49.662345 (+ 3166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:49.663049 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:49.663054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:49.663058 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:49.663172 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:49.663590 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:49.663595 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:49.663598 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:49.663856 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:33:49.664017 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:49.664408 (+ 391us) server_negotiation.cc:299] Negotiation successful
0924 11:33:49.664789 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":167,"server-negotiator.queue_time_us":737,"thread_start_us":584,"threads_started":1}
I20260924 11:33:49.671104 8118 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:52800:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 11:33:49.672644 8118 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:33:49.679744 8256 tablet_service.cc:1467] Processing CreateTablet for tablet 370cc3d79fd04ee4beb02a551c938de0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c74a4b2b85f4406880fb7ecad256fd15]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 11:33:49.679955 8395 tablet_service.cc:1467] Processing CreateTablet for tablet 370cc3d79fd04ee4beb02a551c938de0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c74a4b2b85f4406880fb7ecad256fd15]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 11:33:49.679955 8534 tablet_service.cc:1467] Processing CreateTablet for tablet 370cc3d79fd04ee4beb02a551c938de0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c74a4b2b85f4406880fb7ecad256fd15]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 11:33:49.680486 8395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 370cc3d79fd04ee4beb02a551c938de0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:33:49.680473 8256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 370cc3d79fd04ee4beb02a551c938de0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:33:49.680586 8534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 370cc3d79fd04ee4beb02a551c938de0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:33:49.688400 8624 tablet_bootstrap.cc:492] T 370cc3d79fd04ee4beb02a551c938de0 P 3aae29554c0f426299acdeed32fe630b: Bootstrap starting.
I20260924 11:33:49.688473 8625 tablet_bootstrap.cc:492] T 370cc3d79fd04ee4beb02a551c938de0 P c48acea63acb4d6aaf797e3574a4cad3: Bootstrap starting.
I20260924 11:33:49.689980 8624 tablet_bootstrap.cc:654] T 370cc3d79fd04ee4beb02a551c938de0 P 3aae29554c0f426299acdeed32fe630b: Neither blocks nor log segments found. Creating new log.
I20260924 11:33:49.690994 8625 tablet_bootstrap.cc:654] T 370cc3d79fd04ee4beb02a551c938de0 P c48acea63acb4d6aaf797e3574a4cad3: Neither blocks nor log segments found. Creating new log.
I20260924 11:33:49.692546 8624 tablet_bootstrap.cc:492] T 370cc3d79fd04ee4beb02a551c938de0 P 3aae29554c0f426299acdeed32fe630b: No bootstrap required, opened a new log
I20260924 11:33:49.692706 8624 ts_tablet_manager.cc:1397] T 370cc3d79fd04ee4beb02a551c938de0 P 3aae29554c0f426299acdeed32fe630b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260924 11:33:49.693513 8624 raft_consensus.cc:348] T 370cc3d79fd04ee4beb02a551c938de0 P 3aae29554c0f426299acdeed32fe630b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e9dd80d738487cb5e99a5d55eded82" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46675 } } peers { permanent_uuid: "3aae29554c0f426299acdeed32fe630b" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36373 } } peers { permanent_uuid: "c48acea63acb4d6aaf797e3574a4cad3" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43581 } }
I20260924 11:33:49.693778 8624 raft_consensus.cc:374] T 370cc3d79fd04ee4beb02a551c938de0 P 3aae29554c0f426299acdeed32fe630b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:33:49.693857 8624 raft_consensus.cc:729] T 370cc3d79fd04ee4beb02a551c938de0 P 3aae29554c0f426299acdeed32fe630b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aae29554c0f426299acdeed32fe630b, State: Initialized, Role: FOLLOWER
I20260924 11:33:49.694157 8624 consensus_queue.cc:260] T 370cc3d79fd04ee4beb02a551c938de0 P 3aae29554c0f426299acdeed32fe630b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e9dd80d738487cb5e99a5d55eded82" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46675 } } peers { permanent_uuid: "3aae29554c0f426299acdeed32fe630b" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36373 } } peers { permanent_uuid: "c48acea63acb4d6aaf797e3574a4cad3" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43581 } }
I20260924 11:33:49.694566 8625 tablet_bootstrap.cc:492] T 370cc3d79fd04ee4beb02a551c938de0 P c48acea63acb4d6aaf797e3574a4cad3: No bootstrap required, opened a new log
I20260924 11:33:49.694675 8630 tablet_replica.cc:439] T 370cc3d79fd04ee4beb02a551c938de0 P 3aae29554c0f426299acdeed32fe630b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e9dd80d738487cb5e99a5d55eded82" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46675 } } peers { permanent_uuid: "3aae29554c0f426299acdeed32fe630b" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36373 } } peers { permanent_uuid: "c48acea63acb4d6aaf797e3574a4cad3" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43581 } } }
I20260924 11:33:49.694763 8625 ts_tablet_manager.cc:1397] T 370cc3d79fd04ee4beb02a551c938de0 P c48acea63acb4d6aaf797e3574a4cad3: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260924 11:33:49.694934 8630 tablet_replica.cc:442] T 370cc3d79fd04ee4beb02a551c938de0 P 3aae29554c0f426299acdeed32fe630b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:33:49.695648 8625 raft_consensus.cc:348] T 370cc3d79fd04ee4beb02a551c938de0 P c48acea63acb4d6aaf797e3574a4cad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e9dd80d738487cb5e99a5d55eded82" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46675 } } peers { permanent_uuid: "3aae29554c0f426299acdeed32fe630b" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36373 } } peers { permanent_uuid: "c48acea63acb4d6aaf797e3574a4cad3" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43581 } }
I20260924 11:33:49.695847 8624 ts_tablet_manager.cc:1428] T 370cc3d79fd04ee4beb02a551c938de0 P 3aae29554c0f426299acdeed32fe630b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 11:33:49.695947 8625 raft_consensus.cc:374] T 370cc3d79fd04ee4beb02a551c938de0 P c48acea63acb4d6aaf797e3574a4cad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:33:49.696157 8625 raft_consensus.cc:729] T 370cc3d79fd04ee4beb02a551c938de0 P c48acea63acb4d6aaf797e3574a4cad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c48acea63acb4d6aaf797e3574a4cad3, State: Initialized, Role: FOLLOWER
I20260924 11:33:49.696434 8625 consensus_queue.cc:260] T 370cc3d79fd04ee4beb02a551c938de0 P c48acea63acb4d6aaf797e3574a4cad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e9dd80d738487cb5e99a5d55eded82" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46675 } } peers { permanent_uuid: "3aae29554c0f426299acdeed32fe630b" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36373 } } peers { permanent_uuid: "c48acea63acb4d6aaf797e3574a4cad3" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43581 } }
I20260924 11:33:49.698920 8642 tablet_replica.cc:439] T 370cc3d79fd04ee4beb02a551c938de0 P c48acea63acb4d6aaf797e3574a4cad3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e9dd80d738487cb5e99a5d55eded82" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46675 } } peers { permanent_uuid: "3aae29554c0f426299acdeed32fe630b" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36373 } } peers { permanent_uuid: "c48acea63acb4d6aaf797e3574a4cad3" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43581 } } }
I20260924 11:33:49.699285 8642 tablet_replica.cc:442] T 370cc3d79fd04ee4beb02a551c938de0 P c48acea63acb4d6aaf797e3574a4cad3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:33:49.699785 8625 ts_tablet_manager.cc:1428] T 370cc3d79fd04ee4beb02a551c938de0 P c48acea63acb4d6aaf797e3574a4cad3: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260924 11:33:49.702248 8623 tablet_bootstrap.cc:492] T 370cc3d79fd04ee4beb02a551c938de0 P e7e9dd80d738487cb5e99a5d55eded82: Bootstrap starting.
I20260924 11:33:49.704273 8623 tablet_bootstrap.cc:654] T 370cc3d79fd04ee4beb02a551c938de0 P e7e9dd80d738487cb5e99a5d55eded82: Neither blocks nor log segments found. Creating new log.
I20260924 11:33:49.717113 8623 tablet_bootstrap.cc:492] T 370cc3d79fd04ee4beb02a551c938de0 P e7e9dd80d738487cb5e99a5d55eded82: No bootstrap required, opened a new log
I20260924 11:33:49.717320 8623 ts_tablet_manager.cc:1397] T 370cc3d79fd04ee4beb02a551c938de0 P e7e9dd80d738487cb5e99a5d55eded82: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.011s
I20260924 11:33:49.718243 8623 raft_consensus.cc:348] T 370cc3d79fd04ee4beb02a551c938de0 P e7e9dd80d738487cb5e99a5d55eded82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e9dd80d738487cb5e99a5d55eded82" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46675 } } peers { permanent_uuid: "3aae29554c0f426299acdeed32fe630b" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36373 } } peers { permanent_uuid: "c48acea63acb4d6aaf797e3574a4cad3" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43581 } }
I20260924 11:33:49.718524 8623 raft_consensus.cc:374] T 370cc3d79fd04ee4beb02a551c938de0 P e7e9dd80d738487cb5e99a5d55eded82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:33:49.718627 8623 raft_consensus.cc:729] T 370cc3d79fd04ee4beb02a551c938de0 P e7e9dd80d738487cb5e99a5d55eded82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7e9dd80d738487cb5e99a5d55eded82, State: Initialized, Role: FOLLOWER
I20260924 11:33:49.718904 8623 consensus_queue.cc:260] T 370cc3d79fd04ee4beb02a551c938de0 P e7e9dd80d738487cb5e99a5d55eded82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e9dd80d738487cb5e99a5d55eded82" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46675 } } peers { permanent_uuid: "3aae29554c0f426299acdeed32fe630b" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36373 } } peers { permanent_uuid: "c48acea63acb4d6aaf797e3574a4cad3" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43581 } }
I20260924 11:33:49.719501 8631 tablet_replica.cc:439] T 370cc3d79fd04ee4beb02a551c938de0 P e7e9dd80d738487cb5e99a5d55eded82: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e9dd80d738487cb5e99a5d55eded82" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46675 } } peers { permanent_uuid: "3aae29554c0f426299acdeed32fe630b" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36373 } } peers { permanent_uuid: "c48acea63acb4d6aaf797e3574a4cad3" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43581 } } }
I20260924 11:33:49.719853 8631 tablet_replica.cc:442] T 370cc3d79fd04ee4beb02a551c938de0 P e7e9dd80d738487cb5e99a5d55eded82: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:33:49.720386 8623 ts_tablet_manager.cc:1428] T 370cc3d79fd04ee4beb02a551c938de0 P e7e9dd80d738487cb5e99a5d55eded82: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 11:33:49.790858 8630 raft_consensus.cc:482] T 370cc3d79fd04ee4beb02a551c938de0 P 3aae29554c0f426299acdeed32fe630b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:33:49.791145 8630 raft_consensus.cc:504] T 370cc3d79fd04ee4beb02a551c938de0 P 3aae29554c0f426299acdeed32fe630b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e9dd80d738487cb5e99a5d55eded82" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46675 } } peers { permanent_uuid: "3aae29554c0f426299acdeed32fe630b" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36373 } } peers { permanent_uuid: "c48acea63acb4d6aaf797e3574a4cad3" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43581 } }
I20260924 11:33:49.791903 8630 leader_election.cc:288] T 370cc3d79fd04ee4beb02a551c938de0 P 3aae29554c0f426299acdeed32fe630b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7e9dd80d738487cb5e99a5d55eded82 (127.6.212.1:46675), c48acea63acb4d6aaf797e3574a4cad3 (127.6.212.3:43581)
I20260924 11:33:49.792464 8554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370cc3d79fd04ee4beb02a551c938de0" candidate_uuid: "3aae29554c0f426299acdeed32fe630b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c48acea63acb4d6aaf797e3574a4cad3" is_pre_election: true
I20260924 11:33:49.792486 8280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370cc3d79fd04ee4beb02a551c938de0" candidate_uuid: "3aae29554c0f426299acdeed32fe630b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7e9dd80d738487cb5e99a5d55eded82" is_pre_election: true
I20260924 11:33:49.792708 8554 raft_consensus.cc:2454] T 370cc3d79fd04ee4beb02a551c938de0 P c48acea63acb4d6aaf797e3574a4cad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aae29554c0f426299acdeed32fe630b in term 0.
I20260924 11:33:49.792829 8280 raft_consensus.cc:2454] T 370cc3d79fd04ee4beb02a551c938de0 P e7e9dd80d738487cb5e99a5d55eded82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aae29554c0f426299acdeed32fe630b in term 0.
I20260924 11:33:49.793175 8347 leader_election.cc:302] T 370cc3d79fd04ee4beb02a551c938de0 P 3aae29554c0f426299acdeed32fe630b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aae29554c0f426299acdeed32fe630b, c48acea63acb4d6aaf797e3574a4cad3; no voters:
I20260924 11:33:49.793439 8630 raft_consensus.cc:2781] T 370cc3d79fd04ee4beb02a551c938de0 P 3aae29554c0f426299acdeed32fe630b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:33:49.793540 8630 raft_consensus.cc:482] T 370cc3d79fd04ee4beb02a551c938de0 P 3aae29554c0f426299acdeed32fe630b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:33:49.793699 8630 raft_consensus.cc:3037] T 370cc3d79fd04ee4beb02a551c938de0 P 3aae29554c0f426299acdeed32fe630b [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:33:49.796149 8630 raft_consensus.cc:504] T 370cc3d79fd04ee4beb02a551c938de0 P 3aae29554c0f426299acdeed32fe630b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e9dd80d738487cb5e99a5d55eded82" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46675 } } peers { permanent_uuid: "3aae29554c0f426299acdeed32fe630b" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36373 } } peers { permanent_uuid: "c48acea63acb4d6aaf797e3574a4cad3" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43581 } }
I20260924 11:33:49.796720 8630 leader_election.cc:288] T 370cc3d79fd04ee4beb02a551c938de0 P 3aae29554c0f426299acdeed32fe630b [CANDIDATE]: Term 1 election: Requested vote from peers e7e9dd80d738487cb5e99a5d55eded82 (127.6.212.1:46675), c48acea63acb4d6aaf797e3574a4cad3 (127.6.212.3:43581)
I20260924 11:33:49.797181 8554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370cc3d79fd04ee4beb02a551c938de0" candidate_uuid: "3aae29554c0f426299acdeed32fe630b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c48acea63acb4d6aaf797e3574a4cad3"
I20260924 11:33:49.797478 8554 raft_consensus.cc:3037] T 370cc3d79fd04ee4beb02a551c938de0 P c48acea63acb4d6aaf797e3574a4cad3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:33:49.799539 8554 raft_consensus.cc:2454] T 370cc3d79fd04ee4beb02a551c938de0 P c48acea63acb4d6aaf797e3574a4cad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aae29554c0f426299acdeed32fe630b in term 1.
I20260924 11:33:49.799904 8347 leader_election.cc:302] T 370cc3d79fd04ee4beb02a551c938de0 P 3aae29554c0f426299acdeed32fe630b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aae29554c0f426299acdeed32fe630b, c48acea63acb4d6aaf797e3574a4cad3; no voters:
I20260924 11:33:49.799896 8280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "370cc3d79fd04ee4beb02a551c938de0" candidate_uuid: "3aae29554c0f426299acdeed32fe630b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7e9dd80d738487cb5e99a5d55eded82"
I20260924 11:33:49.800144 8280 raft_consensus.cc:3037] T 370cc3d79fd04ee4beb02a551c938de0 P e7e9dd80d738487cb5e99a5d55eded82 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:33:49.800170 8630 raft_consensus.cc:2781] T 370cc3d79fd04ee4beb02a551c938de0 P 3aae29554c0f426299acdeed32fe630b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:33:49.800338 8630 raft_consensus.cc:686] T 370cc3d79fd04ee4beb02a551c938de0 P 3aae29554c0f426299acdeed32fe630b [term 1 LEADER]: Becoming Leader. State: Replica: 3aae29554c0f426299acdeed32fe630b, State: Running, Role: LEADER
I20260924 11:33:49.800606 8630 consensus_queue.cc:237] T 370cc3d79fd04ee4beb02a551c938de0 P 3aae29554c0f426299acdeed32fe630b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e9dd80d738487cb5e99a5d55eded82" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46675 } } peers { permanent_uuid: "3aae29554c0f426299acdeed32fe630b" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36373 } } peers { permanent_uuid: "c48acea63acb4d6aaf797e3574a4cad3" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43581 } }
I20260924 11:33:49.803129 8655 tablet_replica.cc:439] T 370cc3d79fd04ee4beb02a551c938de0 P 3aae29554c0f426299acdeed32fe630b: TxnStatusTablet state changed. Reason: New leader 3aae29554c0f426299acdeed32fe630b. Latest consensus state: current_term: 1 leader_uuid: "3aae29554c0f426299acdeed32fe630b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e9dd80d738487cb5e99a5d55eded82" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46675 } } peers { permanent_uuid: "3aae29554c0f426299acdeed32fe630b" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36373 } } peers { permanent_uuid: "c48acea63acb4d6aaf797e3574a4cad3" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43581 } } }
I20260924 11:33:49.803452 8655 tablet_replica.cc:442] T 370cc3d79fd04ee4beb02a551c938de0 P 3aae29554c0f426299acdeed32fe630b: This TxnStatusTablet replica's current role is: LEADER
I20260924 11:33:49.803418 8280 raft_consensus.cc:2454] T 370cc3d79fd04ee4beb02a551c938de0 P e7e9dd80d738487cb5e99a5d55eded82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aae29554c0f426299acdeed32fe630b in term 1.
I20260924 11:33:49.804976 8668 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 11:33:49.805260 8668 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 11:33:49.805259 8118 catalog_manager.cc:5495] T 370cc3d79fd04ee4beb02a551c938de0 P 3aae29554c0f426299acdeed32fe630b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aae29554c0f426299acdeed32fe630b (127.6.212.2). New cstate: current_term: 1 leader_uuid: "3aae29554c0f426299acdeed32fe630b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e9dd80d738487cb5e99a5d55eded82" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aae29554c0f426299acdeed32fe630b" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c48acea63acb4d6aaf797e3574a4cad3" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43581 } health_report { overall_health: UNKNOWN } } }
I20260924 11:33:49.838502 8671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:49.829810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54106 (local address 127.6.212.2:36373)
0924 11:33:49.830599 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:49.830605 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:49.830630 (+ 25us) server_negotiation.cc:407] Connection header received
0924 11:33:49.830730 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:49.830735 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:49.830831 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:49.830998 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:33:49.831489 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:49.832793 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:49.835476 (+ 2683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:49.836200 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:49.836329 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:49.836963 (+ 634us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:33:49.837793 (+ 830us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:33:49.837952 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:49.838050 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 11:33:49.838139 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":414,"threads_started":1}
I20260924 11:33:49.838548 8662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:49.829648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.2:36373 (local address 127.0.0.1:54106)
0924 11:33:49.829881 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 11:33:49.829904 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 11:33:49.830054 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:33:49.831061 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:33:49.831066 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:33:49.831082 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 11:33:49.831291 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:49.831303 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:49.832966 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:49.832973 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:49.835297 (+ 2324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:49.835314 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:49.836383 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:49.836390 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:49.836611 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:49.836690 (+ 79us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 11:33:49.837912 (+ 1222us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 11:33:49.837934 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 11:33:49.838029 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 11:33:49.838201 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 11:33:49.844995 8280 raft_consensus.cc:1261] T 370cc3d79fd04ee4beb02a551c938de0 P e7e9dd80d738487cb5e99a5d55eded82 [term 1 FOLLOWER]: Refusing update from remote peer 3aae29554c0f426299acdeed32fe630b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:33:49.845201 8554 raft_consensus.cc:1261] T 370cc3d79fd04ee4beb02a551c938de0 P c48acea63acb4d6aaf797e3574a4cad3 [term 1 FOLLOWER]: Refusing update from remote peer 3aae29554c0f426299acdeed32fe630b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:33:49.845791 8655 consensus_queue.cc:1035] T 370cc3d79fd04ee4beb02a551c938de0 P 3aae29554c0f426299acdeed32fe630b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7e9dd80d738487cb5e99a5d55eded82" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:33:49.846140 8630 consensus_queue.cc:1035] T 370cc3d79fd04ee4beb02a551c938de0 P 3aae29554c0f426299acdeed32fe630b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c48acea63acb4d6aaf797e3574a4cad3" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:33:49.867233 8631 tablet_replica.cc:439] T 370cc3d79fd04ee4beb02a551c938de0 P e7e9dd80d738487cb5e99a5d55eded82: TxnStatusTablet state changed. Reason: New leader 3aae29554c0f426299acdeed32fe630b. Latest consensus state: current_term: 1 leader_uuid: "3aae29554c0f426299acdeed32fe630b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e9dd80d738487cb5e99a5d55eded82" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46675 } } peers { permanent_uuid: "3aae29554c0f426299acdeed32fe630b" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36373 } } peers { permanent_uuid: "c48acea63acb4d6aaf797e3574a4cad3" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43581 } } }
I20260924 11:33:49.867594 8631 tablet_replica.cc:442] T 370cc3d79fd04ee4beb02a551c938de0 P e7e9dd80d738487cb5e99a5d55eded82: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:33:49.867453 8642 tablet_replica.cc:439] T 370cc3d79fd04ee4beb02a551c938de0 P c48acea63acb4d6aaf797e3574a4cad3: TxnStatusTablet state changed. Reason: New leader 3aae29554c0f426299acdeed32fe630b. Latest consensus state: current_term: 1 leader_uuid: "3aae29554c0f426299acdeed32fe630b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e9dd80d738487cb5e99a5d55eded82" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46675 } } peers { permanent_uuid: "3aae29554c0f426299acdeed32fe630b" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36373 } } peers { permanent_uuid: "c48acea63acb4d6aaf797e3574a4cad3" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43581 } } }
I20260924 11:33:49.867743 8642 tablet_replica.cc:442] T 370cc3d79fd04ee4beb02a551c938de0 P c48acea63acb4d6aaf797e3574a4cad3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:33:49.873627 8655 tablet_replica.cc:439] T 370cc3d79fd04ee4beb02a551c938de0 P 3aae29554c0f426299acdeed32fe630b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3aae29554c0f426299acdeed32fe630b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e9dd80d738487cb5e99a5d55eded82" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46675 } } peers { permanent_uuid: "3aae29554c0f426299acdeed32fe630b" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36373 } } peers { permanent_uuid: "c48acea63acb4d6aaf797e3574a4cad3" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43581 } } }
I20260924 11:33:49.874068 8655 tablet_replica.cc:442] T 370cc3d79fd04ee4beb02a551c938de0 P 3aae29554c0f426299acdeed32fe630b: This TxnStatusTablet replica's current role is: LEADER
I20260924 11:33:49.878710 8631 tablet_replica.cc:439] T 370cc3d79fd04ee4beb02a551c938de0 P e7e9dd80d738487cb5e99a5d55eded82: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3aae29554c0f426299acdeed32fe630b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e9dd80d738487cb5e99a5d55eded82" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46675 } } peers { permanent_uuid: "3aae29554c0f426299acdeed32fe630b" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36373 } } peers { permanent_uuid: "c48acea63acb4d6aaf797e3574a4cad3" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43581 } } }
I20260924 11:33:49.878974 8631 tablet_replica.cc:442] T 370cc3d79fd04ee4beb02a551c938de0 P e7e9dd80d738487cb5e99a5d55eded82: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:33:49.884461 8642 tablet_replica.cc:439] T 370cc3d79fd04ee4beb02a551c938de0 P c48acea63acb4d6aaf797e3574a4cad3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3aae29554c0f426299acdeed32fe630b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e9dd80d738487cb5e99a5d55eded82" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46675 } } peers { permanent_uuid: "3aae29554c0f426299acdeed32fe630b" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36373 } } peers { permanent_uuid: "c48acea63acb4d6aaf797e3574a4cad3" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43581 } } }
I20260924 11:33:49.884730 8642 tablet_replica.cc:442] T 370cc3d79fd04ee4beb02a551c938de0 P c48acea63acb4d6aaf797e3574a4cad3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:33:49.882727 8630 tablet_replica.cc:439] T 370cc3d79fd04ee4beb02a551c938de0 P 3aae29554c0f426299acdeed32fe630b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3aae29554c0f426299acdeed32fe630b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7e9dd80d738487cb5e99a5d55eded82" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46675 } } peers { permanent_uuid: "3aae29554c0f426299acdeed32fe630b" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36373 } } peers { permanent_uuid: "c48acea63acb4d6aaf797e3574a4cad3" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43581 } } }
I20260924 11:33:49.887082 8630 tablet_replica.cc:442] T 370cc3d79fd04ee4beb02a551c938de0 P 3aae29554c0f426299acdeed32fe630b: This TxnStatusTablet replica's current role is: LEADER
I20260924 11:33:49.894840 8671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:49.888544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54108 (local address 127.6.212.2:36373)
0924 11:33:49.888734 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:49.888741 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:49.889321 (+ 580us) server_negotiation.cc:407] Connection header received
0924 11:33:49.889531 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:49.889537 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:49.889611 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:49.889674 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:33:49.890375 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:49.891414 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:49.892985 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:49.893483 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:49.893647 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:49.894068 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:33:49.894172 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:33:49.894286 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:49.894443 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 11:33:49.894525 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 11:33:49.913151 8671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:49.906852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:45139 (local address 127.6.212.2:36373)
0924 11:33:49.907051 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:49.907057 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:49.907843 (+ 786us) server_negotiation.cc:407] Connection header received
0924 11:33:49.908114 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:49.908119 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:49.908237 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:49.908315 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:33:49.908991 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:49.909895 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:49.911321 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:49.911804 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:49.911945 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:49.912358 (+ 413us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:33:49.912458 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:33:49.912556 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:49.912653 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 11:33:49.912770 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":51}
I20260924 11:33:49.913265 8684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:49.906902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.2:36373 (local address 127.6.212.2:45139)
0924 11:33:49.907744 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0924 11:33:49.907765 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 11:33:49.907941 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:33:49.908478 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:33:49.908483 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:33:49.908510 (+ 27us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 11:33:49.908854 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:49.908864 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:49.910007 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:49.910015 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:49.911180 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:49.911192 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:49.911988 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:49.911996 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:49.912137 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:49.912184 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 11:33:49.912555 (+ 371us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 11:33:49.912566 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 11:33:49.912640 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 11:33:49.912822 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":22,"thread_start_us":312,"threads_started":1}
I20260924 11:33:49.922663 8554 raft_consensus.cc:1261] T 9501ff7764b4445fb430ff283a8df0b1 P c48acea63acb4d6aaf797e3574a4cad3 [term 1 FOLLOWER]: Refusing update from remote peer 3aae29554c0f426299acdeed32fe630b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:33:49.922966 8279 raft_consensus.cc:1261] T 9501ff7764b4445fb430ff283a8df0b1 P e7e9dd80d738487cb5e99a5d55eded82 [term 1 FOLLOWER]: Refusing update from remote peer 3aae29554c0f426299acdeed32fe630b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:33:49.923245 8685 consensus_queue.cc:1035] T 9501ff7764b4445fb430ff283a8df0b1 P 3aae29554c0f426299acdeed32fe630b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c48acea63acb4d6aaf797e3574a4cad3" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:33:49.924238 8630 consensus_queue.cc:1035] T 9501ff7764b4445fb430ff283a8df0b1 P 3aae29554c0f426299acdeed32fe630b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7e9dd80d738487cb5e99a5d55eded82" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:33:50.020969 8119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:33:50.021184 8119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:33:50.024823 8119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 092352e3835147d59f8e8772a70a9475: Sending DeleteTablet for 3 replicas of tablet 9501ff7764b4445fb430ff283a8df0b1
I20260924 11:33:50.025808 8534 tablet_service.cc:1514] Processing DeleteTablet for tablet 9501ff7764b4445fb430ff283a8df0b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:33:50 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:53916
I20260924 11:33:50.025832 8256 tablet_service.cc:1514] Processing DeleteTablet for tablet 9501ff7764b4445fb430ff283a8df0b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:33:50 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:35354
I20260924 11:33:50.025862 8394 tablet_service.cc:1514] Processing DeleteTablet for tablet 9501ff7764b4445fb430ff283a8df0b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:33:50 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:54092
I20260924 11:33:50.026554 8691 tablet_replica.cc:330] T 9501ff7764b4445fb430ff283a8df0b1 P c48acea63acb4d6aaf797e3574a4cad3: stopping tablet replica
I20260924 11:33:50.026819 8691 raft_consensus.cc:2229] T 9501ff7764b4445fb430ff283a8df0b1 P c48acea63acb4d6aaf797e3574a4cad3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:33:50.027122 8691 raft_consensus.cc:2258] T 9501ff7764b4445fb430ff283a8df0b1 P c48acea63acb4d6aaf797e3574a4cad3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:33:50.028514 8691 ts_tablet_manager.cc:1905] T 9501ff7764b4445fb430ff283a8df0b1 P c48acea63acb4d6aaf797e3574a4cad3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:33:50.036127 8693 tablet_replica.cc:330] T 9501ff7764b4445fb430ff283a8df0b1 P 3aae29554c0f426299acdeed32fe630b: stopping tablet replica
I20260924 11:33:50.036592 8693 raft_consensus.cc:2229] T 9501ff7764b4445fb430ff283a8df0b1 P 3aae29554c0f426299acdeed32fe630b [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:33:50.037278 8691 ts_tablet_manager.cc:1918] T 9501ff7764b4445fb430ff283a8df0b1 P c48acea63acb4d6aaf797e3574a4cad3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 11:33:50.037472 8691 log.cc:1196] T 9501ff7764b4445fb430ff283a8df0b1 P c48acea63acb4d6aaf797e3574a4cad3: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-2/wal/wals/9501ff7764b4445fb430ff283a8df0b1
I20260924 11:33:50.037643 8693 raft_consensus.cc:2258] T 9501ff7764b4445fb430ff283a8df0b1 P 3aae29554c0f426299acdeed32fe630b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:33:50.037952 8691 ts_tablet_manager.cc:1939] T 9501ff7764b4445fb430ff283a8df0b1 P c48acea63acb4d6aaf797e3574a4cad3: Deleting consensus metadata
I20260924 11:33:50.039215 8104 catalog_manager.cc:4841] TS c48acea63acb4d6aaf797e3574a4cad3 (127.6.212.3:43581): tablet 9501ff7764b4445fb430ff283a8df0b1 (table test-table [id=07cdb4a3f73a4d22b0e4fc9584b491f1]) successfully deleted
I20260924 11:33:50.040107 8692 tablet_replica.cc:330] T 9501ff7764b4445fb430ff283a8df0b1 P e7e9dd80d738487cb5e99a5d55eded82: stopping tablet replica
I20260924 11:33:50.040787 8693 ts_tablet_manager.cc:1905] T 9501ff7764b4445fb430ff283a8df0b1 P 3aae29554c0f426299acdeed32fe630b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:33:50.041185 8692 raft_consensus.cc:2229] T 9501ff7764b4445fb430ff283a8df0b1 P e7e9dd80d738487cb5e99a5d55eded82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:33:50.043656 8692 raft_consensus.cc:2258] T 9501ff7764b4445fb430ff283a8df0b1 P e7e9dd80d738487cb5e99a5d55eded82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:33:50.044951 8692 ts_tablet_manager.cc:1905] T 9501ff7764b4445fb430ff283a8df0b1 P e7e9dd80d738487cb5e99a5d55eded82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:33:50.052840 8692 ts_tablet_manager.cc:1918] T 9501ff7764b4445fb430ff283a8df0b1 P e7e9dd80d738487cb5e99a5d55eded82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 11:33:50.053074 8692 log.cc:1196] T 9501ff7764b4445fb430ff283a8df0b1 P e7e9dd80d738487cb5e99a5d55eded82: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-0/wal/wals/9501ff7764b4445fb430ff283a8df0b1
I20260924 11:33:50.053610 8693 ts_tablet_manager.cc:1918] T 9501ff7764b4445fb430ff283a8df0b1 P 3aae29554c0f426299acdeed32fe630b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 11:33:50.053647 8692 ts_tablet_manager.cc:1939] T 9501ff7764b4445fb430ff283a8df0b1 P e7e9dd80d738487cb5e99a5d55eded82: Deleting consensus metadata
I20260924 11:33:50.053756 8693 log.cc:1196] T 9501ff7764b4445fb430ff283a8df0b1 P 3aae29554c0f426299acdeed32fe630b: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790249617057452-6992-0/minicluster-data/ts-1/wal/wals/9501ff7764b4445fb430ff283a8df0b1
I20260924 11:33:50.054214 8693 ts_tablet_manager.cc:1939] T 9501ff7764b4445fb430ff283a8df0b1 P 3aae29554c0f426299acdeed32fe630b: Deleting consensus metadata
I20260924 11:33:50.055064 8105 catalog_manager.cc:4841] TS e7e9dd80d738487cb5e99a5d55eded82 (127.6.212.1:46675): tablet 9501ff7764b4445fb430ff283a8df0b1 (table test-table [id=07cdb4a3f73a4d22b0e4fc9584b491f1]) successfully deleted
I20260924 11:33:50.055682 8106 catalog_manager.cc:4841] TS 3aae29554c0f426299acdeed32fe630b (127.6.212.2:36373): tablet 9501ff7764b4445fb430ff283a8df0b1 (table test-table [id=07cdb4a3f73a4d22b0e4fc9584b491f1]) successfully deleted
Sep 24 11:33:50 dist-test-slave-cq7c krb5kdc[8074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249628, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.212.1@KRBTEST.COM
I20260924 11:33:50.063472 8620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:50.045103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35356 (local address 127.6.212.1:46675)
0924 11:33:50.045274 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:50.045282 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:50.046399 (+ 1117us) server_negotiation.cc:407] Connection header received
0924 11:33:50.046582 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:50.046588 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:50.046688 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:50.046807 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:33:50.048706 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:50.049771 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:50.051797 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:50.052409 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:50.052519 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:50.057624 (+ 5105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:33:50.057646 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:33:50.057662 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:33:50.057700 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:33:50.060684 (+ 2984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:50.061383 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:50.061397 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:50.061399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:50.061530 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:50.061901 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:50.061910 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:50.061912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:50.062257 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:33:50.062579 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:50.062839 (+ 260us) server_negotiation.cc:299] Negotiation successful
0924 11:33:50.063066 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260924 11:33:50.064117 8321 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35356
I20260924 11:33:50.082633 8663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:50.067648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52814 (local address 127.6.212.62:40625)
0924 11:33:50.067847 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:50.067853 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:50.068069 (+ 216us) server_negotiation.cc:407] Connection header received
0924 11:33:50.068284 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:50.068290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:50.068372 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:50.068485 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:33:50.070355 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:50.071599 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:50.072797 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:50.073472 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:50.073604 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:50.076158 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:33:50.076165 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:33:50.076172 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:33:50.076214 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:33:50.079749 (+ 3535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:50.080450 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:50.080456 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:50.080461 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:50.080615 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:50.081011 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:50.081016 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:50.081020 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:50.081363 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:33:50.081603 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:50.081899 (+ 296us) server_negotiation.cc:299] Negotiation successful
0924 11:33:50.082145 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260924 11:33:50.083519 8119 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52814
I20260924 11:33:50.086593 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 8190
I20260924 11:33:50.100632 8620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:50.087911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35360 (local address 127.6.212.1:46675)
0924 11:33:50.088106 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:50.088112 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:50.088789 (+ 677us) server_negotiation.cc:407] Connection header received
0924 11:33:50.089002 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:50.089007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:50.089070 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:50.089177 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:33:50.090325 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:50.091059 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:50.092329 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:50.092802 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:50.092941 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:50.095163 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:33:50.095169 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:33:50.095172 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:33:50.095202 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:33:50.097857 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:50.098513 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:50.098519 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:50.098523 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:50.098655 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:50.099160 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:50.099165 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:50.099169 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:50.099516 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:33:50.099765 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:50.100083 (+ 318us) server_negotiation.cc:299] Negotiation successful
0924 11:33:50.100285 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260924 11:33:50.101286 8321 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35360
W20260924 11:33:50.101945 6992 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 8190: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:33:50.102133 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 8190
I20260924 11:33:50.125240 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 8330
Sep 24 11:33:50 dist-test-slave-cq7c krb5kdc[8074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249628, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.212.2@KRBTEST.COM
I20260924 11:33:50.143075 8671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:50.126055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54114 (local address 127.6.212.2:36373)
0924 11:33:50.126267 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:50.126279 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:50.126341 (+ 62us) server_negotiation.cc:407] Connection header received
0924 11:33:50.126541 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:50.126545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:50.126613 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:50.126699 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:33:50.128236 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:50.128993 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:50.130882 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:50.131636 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:50.131740 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:50.136850 (+ 5110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:33:50.136867 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:33:50.136883 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:33:50.136922 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:33:50.140153 (+ 3231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:50.140899 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:50.140906 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:50.140909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:50.141067 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:50.141465 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:50.141470 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:50.141473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:50.141861 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:33:50.142215 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:50.142504 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 11:33:50.142734 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260924 11:33:50.143844 8460 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54114
W20260924 11:33:50.144626 6992 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 8330: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:33:50.144848 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 8330
I20260924 11:33:50.164690 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 8469
Sep 24 11:33:50 dist-test-slave-cq7c krb5kdc[8074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249628, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.212.3@KRBTEST.COM
I20260924 11:33:50.180609 8699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:50.165522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53944 (local address 127.6.212.3:43581)
0924 11:33:50.166082 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:50.166086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:50.166103 (+ 17us) server_negotiation.cc:407] Connection header received
0924 11:33:50.166173 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:50.166177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:50.166239 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:50.166337 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:33:50.167635 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:50.168345 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:50.169757 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:50.170262 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:50.170360 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:50.175080 (+ 4720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:33:50.175095 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:33:50.175112 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:33:50.175155 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:33:50.177896 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:50.178646 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:50.178654 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:50.178656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:50.178788 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:50.179134 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:50.179137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:50.179138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:50.179559 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:33:50.179804 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:50.180081 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 11:33:50.180239 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":338,"threads_started":1}
W20260924 11:33:50.181342 8599 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53944
W20260924 11:33:50.182113 6992 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 8469: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:33:50.182288 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 8469
I20260924 11:33:50.203823 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 8087
I20260924 11:33:50.205045 8159 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:33:50.322170 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 8087
2026-09-24T11:33:50Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4793 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:33:50 dist-test-slave-cq7c krb5kdc[8703](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:33:50 dist-test-slave-cq7c krb5kdc[8703](info): set up 2 sockets
Sep 24 11:33:50 dist-test-slave-cq7c krb5kdc[8703](info): commencing operation
krb5kdc: starting...
W20260924 11:33:51.427457 6992 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.007s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Sep 24 11:33:51 dist-test-slave-cq7c krb5kdc[8703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:33:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:33:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.212.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
I20260924 11:33:51.624101 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:38939
--webserver_interface=127.6.212.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:40209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:38939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:33:51.949863 8716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:33:51.950156 8716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:33:51.950243 8716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:33:51.957402 8716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:33:51.957507 8716 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 11:33:51.957547 8716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:33:51.957605 8716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:33:51.957653 8716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:33:51.957706 8716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:33:51.967201 8716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:40209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:38939
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:38939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.212.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:33:51.968569 8716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:33:51.969940 8716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:33:51.981448 8721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:33:51.981400 8722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:33:51.981421 8724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:33:51.982422 8716 server_base.cc:1034] running on GCE node
I20260924 11:33:51.983234 8716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:33:51.984695 8716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:33:51.985976 8716 hybrid_clock.cc:648] HybridClock initialized: now 1790249631985936 us; error 65 us; skew 500 ppm
I20260924 11:33:51.986306 8716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:33:51 dist-test-slave-cq7c krb5kdc[8703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249631, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:33:51.989444 8716 init.cc:377] Logged in from keytab as kudu/127.6.212.62@KRBTEST.COM (short username kudu)
I20260924 11:33:51.990849 8716 webserver.cc:460] Webserver started at http://127.6.212.62:46873/ using document root <none> and password file <none>
I20260924 11:33:51.991561 8716 fs_manager.cc:359] Metadata directory not provided
I20260924 11:33:51.991647 8716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:33:51.991911 8716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:33:51.994674 8716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "fa7becf8bf9b4da3b0b57bc5dcadbc18"
format_stamp: "Formatted at 2026-09-24 11:33:51 on dist-test-slave-cq7c"
server_key: "76b374f0607fb5780de0676ddc656b4c"
server_key_iv: "bbdf3bc42353aca248a9a8d544dd6108"
server_key_version: "encryptionkey@0"
I20260924 11:33:51.995352 8716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "fa7becf8bf9b4da3b0b57bc5dcadbc18"
format_stamp: "Formatted at 2026-09-24 11:33:51 on dist-test-slave-cq7c"
server_key: "76b374f0607fb5780de0676ddc656b4c"
server_key_iv: "bbdf3bc42353aca248a9a8d544dd6108"
server_key_version: "encryptionkey@0"
I20260924 11:33:52.000308 8716 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 11:33:52.003556 8731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:33:52.004856 8716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 11:33:52.005051 8716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "fa7becf8bf9b4da3b0b57bc5dcadbc18"
format_stamp: "Formatted at 2026-09-24 11:33:51 on dist-test-slave-cq7c"
server_key: "76b374f0607fb5780de0676ddc656b4c"
server_key_iv: "bbdf3bc42353aca248a9a8d544dd6108"
server_key_version: "encryptionkey@0"
I20260924 11:33:52.005239 8716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:33:52.016922 8716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:33:52.021129 8716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:33:52.021389 8716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:33:52.042982 8716 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.62:38939
I20260924 11:33:52.043002 8792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.62:38939 every 8 connection(s)
I20260924 11:33:52.044335 8716 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:33:52.048187 8793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:33:52.051213 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 8716
I20260924 11:33:52.051474 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:33:52.052098 6992 external_mini_cluster.cc:1428] Setting key 5c995eda4a559f5227ca4d47f64f4166
I20260924 11:33:52.060528 8793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa7becf8bf9b4da3b0b57bc5dcadbc18: Bootstrap starting.
I20260924 11:33:52.064494 8793 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa7becf8bf9b4da3b0b57bc5dcadbc18: Neither blocks nor log segments found. Creating new log.
I20260924 11:33:52.065871 8793 log.cc:824] T 00000000000000000000000000000000 P fa7becf8bf9b4da3b0b57bc5dcadbc18: Log is configured to *not* fsync() on all Append() calls
Sep 24 11:33:52 dist-test-slave-cq7c krb5kdc[8703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:33:52.069648 8793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa7becf8bf9b4da3b0b57bc5dcadbc18: No bootstrap required, opened a new log
I20260924 11:33:52.077039 8796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:52.053976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52466 (local address 127.6.212.62:38939)
0924 11:33:52.054690 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:52.054701 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:52.054742 (+ 41us) server_negotiation.cc:407] Connection header received
0924 11:33:52.056416 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:52.056437 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:52.056843 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:52.057175 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:33:52.058959 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:52.060283 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:52.061944 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:52.062665 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:52.062865 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:52.067981 (+ 5116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:33:52.067989 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:33:52.068005 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:33:52.068044 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:33:52.072353 (+ 4309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:52.073186 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:52.073197 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:52.073211 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:52.073419 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:52.073765 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:52.073769 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:52.073773 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:52.074493 (+ 720us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:33:52.074764 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:52.075131 (+ 367us) server_negotiation.cc:299] Negotiation successful
0924 11:33:52.075620 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":339,"threads_started":1}
I20260924 11:33:52.079856 8793 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa7becf8bf9b4da3b0b57bc5dcadbc18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa7becf8bf9b4da3b0b57bc5dcadbc18" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 38939 } }
I20260924 11:33:52.080170 8793 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa7becf8bf9b4da3b0b57bc5dcadbc18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:33:52.080242 8793 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa7becf8bf9b4da3b0b57bc5dcadbc18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa7becf8bf9b4da3b0b57bc5dcadbc18, State: Initialized, Role: FOLLOWER
I20260924 11:33:52.080709 8793 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa7becf8bf9b4da3b0b57bc5dcadbc18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa7becf8bf9b4da3b0b57bc5dcadbc18" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 38939 } }
I20260924 11:33:52.080904 8793 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa7becf8bf9b4da3b0b57bc5dcadbc18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:33:52.080994 8793 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa7becf8bf9b4da3b0b57bc5dcadbc18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:33:52.081115 8793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa7becf8bf9b4da3b0b57bc5dcadbc18 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:33:52.083194 8793 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa7becf8bf9b4da3b0b57bc5dcadbc18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa7becf8bf9b4da3b0b57bc5dcadbc18" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 38939 } }
I20260924 11:33:52.083746 8793 leader_election.cc:302] T 00000000000000000000000000000000 P fa7becf8bf9b4da3b0b57bc5dcadbc18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa7becf8bf9b4da3b0b57bc5dcadbc18; no voters:
I20260924 11:33:52.084301 8793 leader_election.cc:288] T 00000000000000000000000000000000 P fa7becf8bf9b4da3b0b57bc5dcadbc18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:33:52.084486 8798 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa7becf8bf9b4da3b0b57bc5dcadbc18 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:33:52.085207 8798 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa7becf8bf9b4da3b0b57bc5dcadbc18 [term 1 LEADER]: Becoming Leader. State: Replica: fa7becf8bf9b4da3b0b57bc5dcadbc18, State: Running, Role: LEADER
I20260924 11:33:52.085812 8798 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa7becf8bf9b4da3b0b57bc5dcadbc18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa7becf8bf9b4da3b0b57bc5dcadbc18" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 38939 } }
I20260924 11:33:52.086654 8793 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa7becf8bf9b4da3b0b57bc5dcadbc18 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:33:52.089058 8800 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa7becf8bf9b4da3b0b57bc5dcadbc18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa7becf8bf9b4da3b0b57bc5dcadbc18. Latest consensus state: current_term: 1 leader_uuid: "fa7becf8bf9b4da3b0b57bc5dcadbc18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa7becf8bf9b4da3b0b57bc5dcadbc18" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 38939 } } }
I20260924 11:33:52.089488 8800 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa7becf8bf9b4da3b0b57bc5dcadbc18 [sys.catalog]: This master's current role is: LEADER
I20260924 11:33:52.089849 8799 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa7becf8bf9b4da3b0b57bc5dcadbc18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa7becf8bf9b4da3b0b57bc5dcadbc18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa7becf8bf9b4da3b0b57bc5dcadbc18" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 38939 } } }
I20260924 11:33:52.090315 8799 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa7becf8bf9b4da3b0b57bc5dcadbc18 [sys.catalog]: This master's current role is: LEADER
I20260924 11:33:52.093752 8806 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:33:52.099130 8806 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:33:52.108558 8806 catalog_manager.cc:1347] Generated new cluster ID: a94677d9c3a049ad9e986141cf093ad1
I20260924 11:33:52.108704 8806 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:33:52.121872 8806 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:33:52.123266 8806 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:33:52.132014 8806 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa7becf8bf9b4da3b0b57bc5dcadbc18: Generated new TSK 0
I20260924 11:33:52.132928 8806 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.212.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab.
Entry for principal kudu/127.6.212.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab.
I20260924 11:33:52.187265 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.1:0
--local_ip_for_outbound_sockets=127.6.212.1
--webserver_interface=127.6.212.1
--webserver_port=0
--tserver_master_addrs=127.6.212.62:38939
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab
--principal=kudu/127.6.212.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:40209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:33:52.514922 8819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:33:52.515252 8819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:33:52.515493 8819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:33:52.522811 8819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:33:52.522972 8819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:33:52.523139 8819 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 11:33:52.523226 8819 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 11:33:52.523343 8819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.1
I20260924 11:33:52.533202 8819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:40209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab
--principal=kudu/127.6.212.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.212.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.212.62:38939
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.1
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:33:52.534540 8819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:33:52.535924 8819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:33:52.549188 8825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:33:52.549216 8824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:33:52.549289 8827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:33:52.549944 8819 server_base.cc:1034] running on GCE node
I20260924 11:33:52.550515 8819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:33:52.551538 8819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:33:52.552736 8819 hybrid_clock.cc:648] HybridClock initialized: now 1790249632552693 us; error 39 us; skew 500 ppm
I20260924 11:33:52.553009 8819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:33:52 dist-test-slave-cq7c krb5kdc[8703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249632, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:33:52.556058 8819 init.cc:377] Logged in from keytab as kudu/127.6.212.1@KRBTEST.COM (short username kudu)
I20260924 11:33:52.557574 8819 webserver.cc:460] Webserver started at http://127.6.212.1:44683/ using document root <none> and password file <none>
I20260924 11:33:52.558300 8819 fs_manager.cc:359] Metadata directory not provided
I20260924 11:33:52.558439 8819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:33:52.558771 8819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:33:52.561712 8819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-0/data/instance:
uuid: "5c9aac6a3bab408fb9764d14a9b48903"
format_stamp: "Formatted at 2026-09-24 11:33:52 on dist-test-slave-cq7c"
server_key: "677343e77c70fa8e8217b08e713bb26e"
server_key_iv: "8afb3b43158d931922de9b765584fec9"
server_key_version: "encryptionkey@0"
I20260924 11:33:52.562490 8819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance:
uuid: "5c9aac6a3bab408fb9764d14a9b48903"
format_stamp: "Formatted at 2026-09-24 11:33:52 on dist-test-slave-cq7c"
server_key: "677343e77c70fa8e8217b08e713bb26e"
server_key_iv: "8afb3b43158d931922de9b765584fec9"
server_key_version: "encryptionkey@0"
I20260924 11:33:52.567813 8819 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260924 11:33:52.571585 8834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:33:52.573166 8819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:33:52.573382 8819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-0/wal
uuid: "5c9aac6a3bab408fb9764d14a9b48903"
format_stamp: "Formatted at 2026-09-24 11:33:52 on dist-test-slave-cq7c"
server_key: "677343e77c70fa8e8217b08e713bb26e"
server_key_iv: "8afb3b43158d931922de9b765584fec9"
server_key_version: "encryptionkey@0"
I20260924 11:33:52.573614 8819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:33:52.596872 8819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:33:52.600903 8819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:33:52.601202 8819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:33:52.603816 8819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:33:52.603962 8819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:33:52.604089 8819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:33:52.604178 8819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 11:33:52 dist-test-slave-cq7c krb5kdc[8703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249632, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:33:52.625680 8843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:52.604666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:58013 (local address 127.6.212.62:38939)
0924 11:33:52.605198 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:52.605206 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:52.606432 (+ 1226us) server_negotiation.cc:407] Connection header received
0924 11:33:52.608820 (+ 2388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:52.608826 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:52.608912 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:52.609036 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:33:52.611241 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:52.612038 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:52.613668 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:52.614080 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:52.614211 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:52.618638 (+ 4427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:33:52.618648 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:33:52.618656 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:33:52.618710 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:33:52.622252 (+ 3542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:52.623253 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:52.623259 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:52.623264 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:52.623460 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:52.623913 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:52.623918 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:52.623922 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:52.624201 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:33:52.624361 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:52.625094 (+ 733us) server_negotiation.cc:299] Negotiation successful
0924 11:33:52.625302 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":316,"threads_started":1}
I20260924 11:33:52.626890 8844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:52.605323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:38939 (local address 127.6.212.1:58013)
0924 11:33:52.606251 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0924 11:33:52.606303 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 11:33:52.608507 (+ 2204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:33:52.609322 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:33:52.609337 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:33:52.610135 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:33:52.611053 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:52.611073 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:52.612188 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:52.612194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:52.613522 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:52.613534 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:52.614335 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:52.614342 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:52.614552 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:52.615237 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:33:52.615263 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:33:52.618350 (+ 3087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:33:52.622398 (+ 4048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:33:52.622407 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:33:52.622428 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:33:52.623119 (+ 691us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:33:52.623580 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:33:52.623588 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:33:52.623592 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:33:52.623791 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:33:52.624338 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:33:52.624346 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:33:52.624736 (+ 390us) client_negotiation.cc:769] Sending connection context
0924 11:33:52.625023 (+ 287us) client_negotiation.cc:240] Negotiation successful
0924 11:33:52.625550 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":411,"threads_started":1}
I20260924 11:33:52.651736 8843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:52.634623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:47513 (local address 127.6.212.62:38939)
0924 11:33:52.634804 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:52.634810 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:52.635318 (+ 508us) server_negotiation.cc:407] Connection header received
0924 11:33:52.635620 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:52.635627 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:52.635725 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:52.635832 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:33:52.637609 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:52.638890 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:52.641595 (+ 2705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:52.642387 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:52.642528 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:52.645666 (+ 3138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:33:52.645671 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:33:52.645674 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:33:52.645706 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:33:52.648856 (+ 3150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:52.649805 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:52.649810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:52.649814 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:52.649947 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:52.650337 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:52.650346 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:52.650348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:52.650591 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:33:52.650768 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:52.651155 (+ 387us) server_negotiation.cc:299] Negotiation successful
0924 11:33:52.651390 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 11:33:52.651886 8893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:52.634140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:38939 (local address 127.6.212.1:47513)
0924 11:33:52.635183 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0924 11:33:52.635217 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 11:33:52.635402 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:33:52.636034 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:33:52.636038 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:33:52.636866 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:33:52.637397 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:52.637413 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:52.639102 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:52.639113 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:52.641360 (+ 2247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:52.641380 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:52.642866 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:52.642874 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:52.643186 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:52.643974 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:33:52.644009 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:33:52.645476 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:33:52.648990 (+ 3514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:33:52.648997 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:33:52.649001 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:33:52.649667 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:33:52.650059 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:33:52.650064 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:33:52.650068 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:33:52.650195 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:33:52.650708 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:33:52.650713 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:33:52.650910 (+ 197us) client_negotiation.cc:769] Sending connection context
0924 11:33:52.651051 (+ 141us) client_negotiation.cc:240] Negotiation successful
0924 11:33:52.651390 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":839,"thread_start_us":676,"threads_started":1}
I20260924 11:33:52.658924 8819 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.1:33699
I20260924 11:33:52.658970 8954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.1:33699 every 8 connection(s)
I20260924 11:33:52.660212 8819 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
I20260924 11:33:52.662114 8955 heartbeater.cc:347] Connected to a master server at 127.6.212.62:38939
I20260924 11:33:52.662290 8955 heartbeater.cc:464] Registering TS with master...
I20260924 11:33:52.662955 8955 heartbeater.cc:511] Master 127.6.212.62:38939 requested a full tablet report, sending...
I20260924 11:33:52.664852 8748 ts_manager.cc:194] Registered new tserver with Master: 5c9aac6a3bab408fb9764d14a9b48903 (127.6.212.1:33699)
I20260924 11:33:52.666294 8748 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.1@KRBTEST.COM'} at 127.6.212.1:58013
I20260924 11:33:52.667037 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 8819
I20260924 11:33:52.667240 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance
I20260924 11:33:52.667887 6992 external_mini_cluster.cc:1428] Setting key 4d5969cd565ad0a4a83d9aa45b119844
WARNING: no policy specified for kudu/127.6.212.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab.
Entry for principal kudu/127.6.212.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab.
I20260924 11:33:52.711272 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.2:0
--local_ip_for_outbound_sockets=127.6.212.2
--webserver_interface=127.6.212.2
--webserver_port=0
--tserver_master_addrs=127.6.212.62:38939
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab
--principal=kudu/127.6.212.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:40209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:33:53.036103 8959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:33:53.036382 8959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:33:53.036581 8959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:33:53.044530 8959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:33:53.044653 8959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:33:53.044785 8959 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 11:33:53.044864 8959 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 11:33:53.044970 8959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.2
I20260924 11:33:53.060073 8959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:40209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab
--principal=kudu/127.6.212.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.212.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.212.62:38939
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.2
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:33:53.061575 8959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:33:53.063185 8959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:33:53.077414 8964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:33:53.077536 8965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:33:53.077374 8967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:33:53.078773 8959 server_base.cc:1034] running on GCE node
I20260924 11:33:53.079591 8959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:33:53.080670 8959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:33:53.081923 8959 hybrid_clock.cc:648] HybridClock initialized: now 1790249633081875 us; error 58 us; skew 500 ppm
I20260924 11:33:53.082278 8959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:33:53 dist-test-slave-cq7c krb5kdc[8703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249633, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:33:53.091236 8959 init.cc:377] Logged in from keytab as kudu/127.6.212.2@KRBTEST.COM (short username kudu)
I20260924 11:33:53.092895 8959 webserver.cc:460] Webserver started at http://127.6.212.2:35085/ using document root <none> and password file <none>
I20260924 11:33:53.093665 8959 fs_manager.cc:359] Metadata directory not provided
I20260924 11:33:53.093813 8959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:33:53.094131 8959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:33:53.097216 8959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-1/data/instance:
uuid: "67e69286416b452a8de92d69237c5b08"
format_stamp: "Formatted at 2026-09-24 11:33:53 on dist-test-slave-cq7c"
server_key: "36e46a12e22a85cfe6b2bfb944ea9381"
server_key_iv: "338081abd32d190dec3cc318489877ea"
server_key_version: "encryptionkey@0"
I20260924 11:33:53.098059 8959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance:
uuid: "67e69286416b452a8de92d69237c5b08"
format_stamp: "Formatted at 2026-09-24 11:33:53 on dist-test-slave-cq7c"
server_key: "36e46a12e22a85cfe6b2bfb944ea9381"
server_key_iv: "338081abd32d190dec3cc318489877ea"
server_key_version: "encryptionkey@0"
I20260924 11:33:53.103560 8959 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 11:33:53.106869 8974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:33:53.108220 8959 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:33:53.108397 8959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-1/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-1/wal
uuid: "67e69286416b452a8de92d69237c5b08"
format_stamp: "Formatted at 2026-09-24 11:33:53 on dist-test-slave-cq7c"
server_key: "36e46a12e22a85cfe6b2bfb944ea9381"
server_key_iv: "338081abd32d190dec3cc318489877ea"
server_key_version: "encryptionkey@0"
I20260924 11:33:53.108618 8959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:33:53.135699 8959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:33:53.139748 8959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:33:53.140009 8959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:33:53.142499 8959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:33:53.142642 8959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:33:53.142755 8959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:33:53.142838 8959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 11:33:53 dist-test-slave-cq7c krb5kdc[8703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249633, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.2@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:33:53.167163 8843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:53.143280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:41619 (local address 127.6.212.62:38939)
0924 11:33:53.143469 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:53.143478 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:53.145202 (+ 1724us) server_negotiation.cc:407] Connection header received
0924 11:33:53.147723 (+ 2521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:53.147732 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:53.147864 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:53.148008 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:33:53.150255 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:53.151172 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:53.153955 (+ 2783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:53.154429 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:53.154574 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:53.160834 (+ 6260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:33:53.160843 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:33:53.160849 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:33:53.160903 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:33:53.164151 (+ 3248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:53.164898 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:53.164903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:53.164907 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:53.165029 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:53.165467 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:53.165472 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:53.165475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:53.165750 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:33:53.165916 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:53.166612 (+ 696us) server_negotiation.cc:299] Negotiation successful
0924 11:33:53.166820 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 11:33:53.168517 8983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:53.144050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:38939 (local address 127.6.212.2:41619)
0924 11:33:53.145019 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0924 11:33:53.145067 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 11:33:53.147270 (+ 2203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:33:53.148228 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:33:53.148237 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:33:53.149088 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:33:53.150077 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:53.150098 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:53.151320 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:53.151325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:53.153789 (+ 2464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:53.153802 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:53.154739 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:53.154751 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:53.155133 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:53.156499 (+ 1366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:33:53.156528 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:33:53.160495 (+ 3967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:33:53.164302 (+ 3807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:33:53.164311 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:33:53.164328 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:33:53.164764 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:33:53.165141 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:33:53.165146 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:33:53.165150 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:33:53.165347 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:33:53.165885 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:33:53.165902 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:33:53.166260 (+ 358us) client_negotiation.cc:769] Sending connection context
0924 11:33:53.166525 (+ 265us) client_negotiation.cc:240] Negotiation successful
0924 11:33:53.167069 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":383,"threads_started":1}
I20260924 11:33:53.191143 9093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.2:45989 every 8 connection(s)
I20260924 11:33:53.191143 8959 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.2:45989
I20260924 11:33:53.192989 8959 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
I20260924 11:33:53.195232 9094 heartbeater.cc:347] Connected to a master server at 127.6.212.62:38939
I20260924 11:33:53.195564 9094 heartbeater.cc:464] Registering TS with master...
I20260924 11:33:53.196319 9094 heartbeater.cc:511] Master 127.6.212.62:38939 requested a full tablet report, sending...
I20260924 11:33:53.196957 8843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:53.178943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:42423 (local address 127.6.212.62:38939)
0924 11:33:53.179161 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:53.179170 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:53.179979 (+ 809us) server_negotiation.cc:407] Connection header received
0924 11:33:53.180210 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:53.180219 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:53.180310 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:53.180448 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:33:53.181995 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:53.183173 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:53.185835 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:53.186712 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:53.186856 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:53.189528 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:33:53.189538 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:33:53.189543 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:33:53.189583 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:33:53.193166 (+ 3583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:53.194137 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:53.194146 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:53.194177 (+ 31us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:53.195073 (+ 896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:53.195564 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:53.195567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:53.195569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:53.195929 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:33:53.196040 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:53.196565 (+ 525us) server_negotiation.cc:299] Negotiation successful
0924 11:33:53.196731 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 11:33:53.197261 9061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:53.179017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:38939 (local address 127.6.212.2:42423)
0924 11:33:53.179842 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0924 11:33:53.179869 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 11:33:53.180052 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:33:53.180485 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:33:53.180491 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:33:53.181253 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:33:53.181796 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:53.181823 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:53.183389 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:53.183396 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:53.185660 (+ 2264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:53.185680 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:53.186969 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:53.186978 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:53.187209 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:53.188041 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:33:53.188068 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:33:53.189319 (+ 1251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:33:53.193322 (+ 4003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:33:53.193326 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:33:53.193329 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:33:53.193984 (+ 655us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:33:53.195220 (+ 1236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:33:53.195225 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:33:53.195229 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:33:53.195429 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:33:53.196055 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:33:53.196059 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:33:53.196254 (+ 195us) client_negotiation.cc:769] Sending connection context
0924 11:33:53.196433 (+ 179us) client_negotiation.cc:240] Negotiation successful
0924 11:33:53.196812 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":463,"threads_started":1}
I20260924 11:33:53.197880 8748 ts_manager.cc:194] Registered new tserver with Master: 67e69286416b452a8de92d69237c5b08 (127.6.212.2:45989)
I20260924 11:33:53.198999 8748 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.2@KRBTEST.COM'} at 127.6.212.2:41619
I20260924 11:33:53.201399 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 8959
I20260924 11:33:53.201573 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance
I20260924 11:33:53.202013 6992 external_mini_cluster.cc:1428] Setting key 1cce4038c800afe5cc9895936ec0b9ab
WARNING: no policy specified for kudu/127.6.212.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab.
Entry for principal kudu/127.6.212.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab.
I20260924 11:33:53.243772 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.3:0
--local_ip_for_outbound_sockets=127.6.212.3
--webserver_interface=127.6.212.3
--webserver_port=0
--tserver_master_addrs=127.6.212.62:38939
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab
--principal=kudu/127.6.212.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:40209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:33:53.611989 9098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:33:53.612303 9098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:33:53.612480 9098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:33:53.619769 9098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:33:53.619869 9098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:33:53.620042 9098 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 11:33:53.620119 9098 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 11:33:53.620229 9098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.3
I20260924 11:33:53.630154 9098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:40209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab
--principal=kudu/127.6.212.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.212.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.212.62:38939
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.3
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:33:53.631520 9098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:33:53.632907 9098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:33:53.646382 9103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:33:53.646382 9106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:33:53.646775 9104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:33:53.647222 9098 server_base.cc:1034] running on GCE node
I20260924 11:33:53.647873 9098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:33:53.648833 9098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:33:53.650027 9098 hybrid_clock.cc:648] HybridClock initialized: now 1790249633649970 us; error 56 us; skew 500 ppm
I20260924 11:33:53.650321 9098 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:33:53 dist-test-slave-cq7c krb5kdc[8703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249633, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:33:53.653544 9098 init.cc:377] Logged in from keytab as kudu/127.6.212.3@KRBTEST.COM (short username kudu)
I20260924 11:33:53.654879 9098 webserver.cc:460] Webserver started at http://127.6.212.3:36879/ using document root <none> and password file <none>
I20260924 11:33:53.655597 9098 fs_manager.cc:359] Metadata directory not provided
I20260924 11:33:53.655719 9098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:33:53.656032 9098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:33:53.659041 9098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-2/data/instance:
uuid: "1d9ae86a5c17417c8980ccfc29b037d0"
format_stamp: "Formatted at 2026-09-24 11:33:53 on dist-test-slave-cq7c"
server_key: "faabfe417f7916beb3d27631dea47111"
server_key_iv: "2c1fb3358b56a6ec29eb16ed550ebf38"
server_key_version: "encryptionkey@0"
I20260924 11:33:53.660043 9098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance:
uuid: "1d9ae86a5c17417c8980ccfc29b037d0"
format_stamp: "Formatted at 2026-09-24 11:33:53 on dist-test-slave-cq7c"
server_key: "faabfe417f7916beb3d27631dea47111"
server_key_iv: "2c1fb3358b56a6ec29eb16ed550ebf38"
server_key_version: "encryptionkey@0"
I20260924 11:33:53.666139 9098 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260924 11:33:53.669090 8955 heartbeater.cc:503] Master 127.6.212.62:38939 was elected leader, sending a full tablet report...
I20260924 11:33:53.669720 9113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:33:53.671339 9098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 11:33:53.671551 9098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-2/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-2/wal
uuid: "1d9ae86a5c17417c8980ccfc29b037d0"
format_stamp: "Formatted at 2026-09-24 11:33:53 on dist-test-slave-cq7c"
server_key: "faabfe417f7916beb3d27631dea47111"
server_key_iv: "2c1fb3358b56a6ec29eb16ed550ebf38"
server_key_version: "encryptionkey@0"
I20260924 11:33:53.671762 9098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:33:53.686687 9098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:33:53.691195 9098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:33:53.691556 9098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:33:53.694410 9098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:33:53.694530 9098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:33:53.694638 9098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:33:53.694728 9098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 11:33:53 dist-test-slave-cq7c krb5kdc[8703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249633, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.3@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:33:53.722906 8843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:53.695109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:60771 (local address 127.6.212.62:38939)
0924 11:33:53.695285 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:53.695290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:53.696904 (+ 1614us) server_negotiation.cc:407] Connection header received
0924 11:33:53.699087 (+ 2183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:53.699093 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:53.699183 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:53.699295 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:33:53.701956 (+ 2661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:53.703029 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:53.706481 (+ 3452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:53.707270 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:53.707447 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:53.714181 (+ 6734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:33:53.714190 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:33:53.714198 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:33:53.714239 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:33:53.718276 (+ 4037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:53.719435 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:53.719444 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:53.719450 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:53.719622 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:53.720340 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:53.720348 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:53.720354 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:53.720780 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:33:53.720946 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:53.722129 (+ 1183us) server_negotiation.cc:299] Negotiation successful
0924 11:33:53.722408 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 11:33:53.724663 9122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:53.695777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:38939 (local address 127.6.212.3:60771)
0924 11:33:53.696723 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0924 11:33:53.696767 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 11:33:53.698805 (+ 2038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:33:53.699670 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:33:53.699691 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:33:53.700645 (+ 954us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:33:53.701711 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:53.701738 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:53.703246 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:53.703255 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:53.706283 (+ 3028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:53.706305 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:53.707584 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:53.707593 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:53.707975 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:53.709726 (+ 1751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:33:53.709810 (+ 84us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:33:53.713832 (+ 4022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:33:53.718431 (+ 4599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:33:53.718460 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:33:53.718484 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:33:53.719249 (+ 765us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:33:53.719770 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:33:53.719777 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:33:53.719783 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:33:53.720182 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:33:53.720965 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:33:53.720980 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:33:53.721543 (+ 563us) client_negotiation.cc:769] Sending connection context
0924 11:33:53.721972 (+ 429us) client_negotiation.cc:240] Negotiation successful
0924 11:33:53.722676 (+ 704us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"thread_start_us":470,"threads_started":1}
I20260924 11:33:53.741640 9098 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.3:38443
I20260924 11:33:53.741745 9232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.3:38443 every 8 connection(s)
I20260924 11:33:53.743297 9098 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
I20260924 11:33:53.744076 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 9098
I20260924 11:33:53.744254 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance
I20260924 11:33:53.744781 6992 external_mini_cluster.cc:1428] Setting key d081d46b55533c9499f85c1bf48e5b3b
I20260924 11:33:53.745672 9233 heartbeater.cc:347] Connected to a master server at 127.6.212.62:38939
I20260924 11:33:53.746546 9233 heartbeater.cc:464] Registering TS with master...
I20260924 11:33:53.747339 9233 heartbeater.cc:511] Master 127.6.212.62:38939 requested a full tablet report, sending...
I20260924 11:33:53.748864 8748 ts_manager.cc:194] Registered new tserver with Master: 1d9ae86a5c17417c8980ccfc29b037d0 (127.6.212.3:38443)
I20260924 11:33:53.749643 8843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:53.732233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:52601 (local address 127.6.212.62:38939)
0924 11:33:53.732402 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:53.732407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:53.733397 (+ 990us) server_negotiation.cc:407] Connection header received
0924 11:33:53.733654 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:53.733658 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:53.733729 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:53.733845 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:33:53.734931 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:53.735945 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:53.738351 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:53.739147 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:53.739316 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:53.741943 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:33:53.741950 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:33:53.741955 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:33:53.741998 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:33:53.746623 (+ 4625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:53.747736 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:53.747741 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:53.747744 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:53.747899 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:53.748265 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:53.748269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:53.748271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:53.748652 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:33:53.748763 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:53.749208 (+ 445us) server_negotiation.cc:299] Negotiation successful
0924 11:33:53.749373 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 11:33:53.749895 9211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:53.732446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:38939 (local address 127.6.212.3:52601)
0924 11:33:53.733273 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0924 11:33:53.733298 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 11:33:53.733465 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:33:53.733880 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:33:53.733884 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:33:53.734490 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:33:53.734763 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:53.734792 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:53.736076 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:53.736084 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:53.738177 (+ 2093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:53.738198 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:53.739531 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:53.739540 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:53.739763 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:53.740489 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:33:53.740520 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:33:53.741750 (+ 1230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:33:53.746965 (+ 5215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:33:53.746974 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:33:53.746978 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:33:53.747576 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:33:53.748011 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:33:53.748020 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:33:53.748024 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:33:53.748170 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:33:53.748769 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:33:53.748774 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:33:53.748969 (+ 195us) client_negotiation.cc:769] Sending connection context
0924 11:33:53.749113 (+ 144us) client_negotiation.cc:240] Negotiation successful
0924 11:33:53.749494 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":468,"threads_started":1}
I20260924 11:33:53.749984 8748 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.3@KRBTEST.COM'} at 127.6.212.3:60771
I20260924 11:33:53.760259 6992 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 11:33:53 dist-test-slave-cq7c krb5kdc[8703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 11:33:53 dist-test-slave-cq7c krb5kdc[8703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:33:53.809118 8843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:53.795594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52470 (local address 127.6.212.62:38939)
0924 11:33:53.795758 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:53.795762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:53.796117 (+ 355us) server_negotiation.cc:407] Connection header received
0924 11:33:53.796329 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:53.796333 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:53.796406 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:53.796512 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:33:53.797831 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:53.798574 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:53.799862 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:53.800284 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:53.800422 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:53.804104 (+ 3682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:33:53.804109 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:33:53.804112 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:33:53.804143 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:33:53.806424 (+ 2281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:53.807052 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:53.807056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:53.807059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:53.807169 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:53.807619 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:53.807624 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:53.807629 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:53.807900 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:33:53.808154 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:53.808476 (+ 322us) server_negotiation.cc:299] Negotiation successful
0924 11:33:53.808702 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 11:33:53.813203 8748 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52470:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:33:53.819839 8748 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:33:53.852932 9249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:53.832334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53276 (local address 127.6.212.1:33699)
0924 11:33:53.842549 (+ 10215us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:53.842563 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:53.844429 (+ 1866us) server_negotiation.cc:407] Connection header received
0924 11:33:53.844539 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:53.844548 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:53.844767 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:53.844961 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:33:53.845641 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:53.847103 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:53.850356 (+ 3253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:53.851731 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:53.851896 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:53.852117 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:53.852402 (+ 285us) server_negotiation.cc:299] Negotiation successful
0924 11:33:53.852573 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9987,"thread_start_us":9828,"threads_started":1}
I20260924 11:33:53.853013 9248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:53.831932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:33699 (local address 127.0.0.1:53276)
0924 11:33:53.844165 (+ 12233us) negotiation.cc:107] Waiting for socket to connect
0924 11:33:53.844183 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:33:53.844309 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:33:53.844948 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:33:53.844952 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:33:53.844977 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:33:53.845462 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:53.845475 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:53.847255 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:53.847266 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:53.850174 (+ 2908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:53.850181 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:53.851952 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:53.851958 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:53.852150 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:53.852206 (+ 56us) client_negotiation.cc:769] Sending connection context
0924 11:33:53.852352 (+ 146us) client_negotiation.cc:240] Negotiation successful
0924 11:33:53.852592 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12107,"mutex_wait_us":202,"thread_start_us":12563,"threads_started":1}
I20260924 11:33:53.853588 9246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:53.831745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.2:45989 (local address 127.0.0.1:42920)
0924 11:33:53.843047 (+ 11302us) negotiation.cc:107] Waiting for socket to connect
0924 11:33:53.843097 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 11:33:53.843374 (+ 277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:33:53.844585 (+ 1211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:33:53.844597 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:33:53.844636 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:33:53.845156 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:53.845176 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:53.847093 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:53.847098 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:53.850167 (+ 3069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:53.850181 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:53.852063 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:53.852069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:53.852249 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:53.852289 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 11:33:53.852351 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 11:33:53.852617 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10962,"mutex_wait_us":182,"thread_start_us":10851,"threads_started":1}
I20260924 11:33:53.854889 9250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:53.834531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.3:38443 (local address 127.0.0.1:51384)
0924 11:33:53.846382 (+ 11851us) negotiation.cc:107] Waiting for socket to connect
0924 11:33:53.846398 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:33:53.846545 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:33:53.847664 (+ 1119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:33:53.847668 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:33:53.847695 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:33:53.848272 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:53.848281 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:53.849686 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:53.849691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:53.852040 (+ 2349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:53.852052 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:53.854074 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:53.854083 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:53.854239 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:53.854269 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 11:33:53.854333 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 11:33:53.854414 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11734,"mutex_wait_us":19,"thread_start_us":11602,"threads_started":1}
I20260924 11:33:53.854967 9251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:53.835131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51384 (local address 127.6.212.3:38443)
0924 11:33:53.845930 (+ 10799us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:53.845943 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:53.846514 (+ 571us) server_negotiation.cc:407] Connection header received
0924 11:33:53.846907 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:53.846917 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:53.847134 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:53.847347 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:33:53.848483 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:53.849540 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:53.852207 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:53.853884 (+ 1677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:53.854008 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:53.854227 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:53.854485 (+ 258us) server_negotiation.cc:299] Negotiation successful
0924 11:33:53.854657 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10581,"thread_start_us":10385,"threads_started":1}
I20260924 11:33:53.857024 8889 tablet_service.cc:1467] Processing CreateTablet for tablet 29dcb6019dcf4d4fae63a9e378c2203a (DEFAULT_TABLE table=test-table [id=092c5a403f544dbaaf79ad874d5dff2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:33:53.858304 8889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29dcb6019dcf4d4fae63a9e378c2203a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:33:53.858249 9163 tablet_service.cc:1467] Processing CreateTablet for tablet 29dcb6019dcf4d4fae63a9e378c2203a (DEFAULT_TABLE table=test-table [id=092c5a403f544dbaaf79ad874d5dff2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:33:53.858814 9247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:53.831914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42920 (local address 127.6.212.2:45989)
0924 11:33:53.832474 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:53.832491 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:53.843614 (+ 11123us) server_negotiation.cc:407] Connection header received
0924 11:33:53.843760 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:53.843771 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:53.844095 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:53.844306 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:33:53.845392 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:53.846930 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:53.850354 (+ 3424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:53.851815 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:53.852803 (+ 988us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:53.853010 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:53.853921 (+ 911us) server_negotiation.cc:299] Negotiation successful
0924 11:33:53.858528 (+ 4607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":551,"threads_started":1}
I20260924 11:33:53.860580 9163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29dcb6019dcf4d4fae63a9e378c2203a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:33:53.862196 9024 tablet_service.cc:1467] Processing CreateTablet for tablet 29dcb6019dcf4d4fae63a9e378c2203a (DEFAULT_TABLE table=test-table [id=092c5a403f544dbaaf79ad874d5dff2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:33:53.863525 9024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29dcb6019dcf4d4fae63a9e378c2203a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:33:53.869884 9252 tablet_bootstrap.cc:492] T 29dcb6019dcf4d4fae63a9e378c2203a P 5c9aac6a3bab408fb9764d14a9b48903: Bootstrap starting.
I20260924 11:33:53.874229 9252 tablet_bootstrap.cc:654] T 29dcb6019dcf4d4fae63a9e378c2203a P 5c9aac6a3bab408fb9764d14a9b48903: Neither blocks nor log segments found. Creating new log.
I20260924 11:33:53.874966 9253 tablet_bootstrap.cc:492] T 29dcb6019dcf4d4fae63a9e378c2203a P 1d9ae86a5c17417c8980ccfc29b037d0: Bootstrap starting.
I20260924 11:33:53.875801 9252 log.cc:824] T 29dcb6019dcf4d4fae63a9e378c2203a P 5c9aac6a3bab408fb9764d14a9b48903: Log is configured to *not* fsync() on all Append() calls
I20260924 11:33:53.878659 9253 tablet_bootstrap.cc:654] T 29dcb6019dcf4d4fae63a9e378c2203a P 1d9ae86a5c17417c8980ccfc29b037d0: Neither blocks nor log segments found. Creating new log.
I20260924 11:33:53.879017 9252 tablet_bootstrap.cc:492] T 29dcb6019dcf4d4fae63a9e378c2203a P 5c9aac6a3bab408fb9764d14a9b48903: No bootstrap required, opened a new log
I20260924 11:33:53.879287 9252 ts_tablet_manager.cc:1397] T 29dcb6019dcf4d4fae63a9e378c2203a P 5c9aac6a3bab408fb9764d14a9b48903: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260924 11:33:53.879778 9255 tablet_bootstrap.cc:492] T 29dcb6019dcf4d4fae63a9e378c2203a P 67e69286416b452a8de92d69237c5b08: Bootstrap starting.
I20260924 11:33:53.879909 9253 log.cc:824] T 29dcb6019dcf4d4fae63a9e378c2203a P 1d9ae86a5c17417c8980ccfc29b037d0: Log is configured to *not* fsync() on all Append() calls
I20260924 11:33:53.882889 9255 tablet_bootstrap.cc:654] T 29dcb6019dcf4d4fae63a9e378c2203a P 67e69286416b452a8de92d69237c5b08: Neither blocks nor log segments found. Creating new log.
I20260924 11:33:53.882900 9253 tablet_bootstrap.cc:492] T 29dcb6019dcf4d4fae63a9e378c2203a P 1d9ae86a5c17417c8980ccfc29b037d0: No bootstrap required, opened a new log
I20260924 11:33:53.883168 9253 ts_tablet_manager.cc:1397] T 29dcb6019dcf4d4fae63a9e378c2203a P 1d9ae86a5c17417c8980ccfc29b037d0: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260924 11:33:53.883922 9255 log.cc:824] T 29dcb6019dcf4d4fae63a9e378c2203a P 67e69286416b452a8de92d69237c5b08: Log is configured to *not* fsync() on all Append() calls
I20260924 11:33:53.886170 9255 tablet_bootstrap.cc:492] T 29dcb6019dcf4d4fae63a9e378c2203a P 67e69286416b452a8de92d69237c5b08: No bootstrap required, opened a new log
I20260924 11:33:53.886415 9255 ts_tablet_manager.cc:1397] T 29dcb6019dcf4d4fae63a9e378c2203a P 67e69286416b452a8de92d69237c5b08: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 11:33:53.890180 9252 raft_consensus.cc:348] T 29dcb6019dcf4d4fae63a9e378c2203a P 5c9aac6a3bab408fb9764d14a9b48903 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } }
I20260924 11:33:53.890636 9252 raft_consensus.cc:374] T 29dcb6019dcf4d4fae63a9e378c2203a P 5c9aac6a3bab408fb9764d14a9b48903 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:33:53.890735 9252 raft_consensus.cc:729] T 29dcb6019dcf4d4fae63a9e378c2203a P 5c9aac6a3bab408fb9764d14a9b48903 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c9aac6a3bab408fb9764d14a9b48903, State: Initialized, Role: FOLLOWER
I20260924 11:33:53.891351 9252 consensus_queue.cc:260] T 29dcb6019dcf4d4fae63a9e378c2203a P 5c9aac6a3bab408fb9764d14a9b48903 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } }
I20260924 11:33:53.893050 9252 ts_tablet_manager.cc:1428] T 29dcb6019dcf4d4fae63a9e378c2203a P 5c9aac6a3bab408fb9764d14a9b48903: Time spent starting tablet: real 0.014s user 0.007s sys 0.005s
I20260924 11:33:53.893905 9253 raft_consensus.cc:348] T 29dcb6019dcf4d4fae63a9e378c2203a P 1d9ae86a5c17417c8980ccfc29b037d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } }
I20260924 11:33:53.894312 9253 raft_consensus.cc:374] T 29dcb6019dcf4d4fae63a9e378c2203a P 1d9ae86a5c17417c8980ccfc29b037d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:33:53.894414 9253 raft_consensus.cc:729] T 29dcb6019dcf4d4fae63a9e378c2203a P 1d9ae86a5c17417c8980ccfc29b037d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d9ae86a5c17417c8980ccfc29b037d0, State: Initialized, Role: FOLLOWER
I20260924 11:33:53.894181 9255 raft_consensus.cc:348] T 29dcb6019dcf4d4fae63a9e378c2203a P 67e69286416b452a8de92d69237c5b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } }
I20260924 11:33:53.894593 9255 raft_consensus.cc:374] T 29dcb6019dcf4d4fae63a9e378c2203a P 67e69286416b452a8de92d69237c5b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:33:53.894704 9255 raft_consensus.cc:729] T 29dcb6019dcf4d4fae63a9e378c2203a P 67e69286416b452a8de92d69237c5b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67e69286416b452a8de92d69237c5b08, State: Initialized, Role: FOLLOWER
I20260924 11:33:53.895061 9253 consensus_queue.cc:260] T 29dcb6019dcf4d4fae63a9e378c2203a P 1d9ae86a5c17417c8980ccfc29b037d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } }
I20260924 11:33:53.895381 9255 consensus_queue.cc:260] T 29dcb6019dcf4d4fae63a9e378c2203a P 67e69286416b452a8de92d69237c5b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } }
I20260924 11:33:53.896250 9233 heartbeater.cc:503] Master 127.6.212.62:38939 was elected leader, sending a full tablet report...
I20260924 11:33:53.896910 9253 ts_tablet_manager.cc:1428] T 29dcb6019dcf4d4fae63a9e378c2203a P 1d9ae86a5c17417c8980ccfc29b037d0: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260924 11:33:53.899101 9094 heartbeater.cc:503] Master 127.6.212.62:38939 was elected leader, sending a full tablet report...
I20260924 11:33:53.899891 9255 ts_tablet_manager.cc:1428] T 29dcb6019dcf4d4fae63a9e378c2203a P 67e69286416b452a8de92d69237c5b08: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260924 11:33:53.909530 9260 raft_consensus.cc:482] T 29dcb6019dcf4d4fae63a9e378c2203a P 67e69286416b452a8de92d69237c5b08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:33:53.909809 9260 raft_consensus.cc:504] T 29dcb6019dcf4d4fae63a9e378c2203a P 67e69286416b452a8de92d69237c5b08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } }
I20260924 11:33:53.911481 9260 leader_election.cc:288] T 29dcb6019dcf4d4fae63a9e378c2203a P 67e69286416b452a8de92d69237c5b08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c9aac6a3bab408fb9764d14a9b48903 (127.6.212.1:33699), 1d9ae86a5c17417c8980ccfc29b037d0 (127.6.212.3:38443)
W20260924 11:33:53.912954 8956 tablet.cc:2369] T 29dcb6019dcf4d4fae63a9e378c2203a P 5c9aac6a3bab408fb9764d14a9b48903: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:33:53.918771 9249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:53.911790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:58565 (local address 127.6.212.1:33699)
0924 11:33:53.912027 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:53.912046 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:53.913381 (+ 1335us) server_negotiation.cc:407] Connection header received
0924 11:33:53.913626 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:53.913632 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:53.913713 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:53.913821 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:33:53.914480 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:53.915237 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:53.917063 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:53.917813 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:53.917949 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:53.918065 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:53.918391 (+ 326us) server_negotiation.cc:299] Negotiation successful
0924 11:33:53.918519 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 11:33:53.918799 9251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:53.912292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:38111 (local address 127.6.212.3:38443)
0924 11:33:53.912600 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:53.912635 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:53.913510 (+ 875us) server_negotiation.cc:407] Connection header received
0924 11:33:53.913702 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:53.913708 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:53.913797 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:53.913885 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:33:53.914630 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:53.915387 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:53.917112 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:53.917826 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:53.917970 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:53.918077 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:53.918447 (+ 370us) server_negotiation.cc:299] Negotiation successful
0924 11:33:53.918564 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 11:33:53.918977 9261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:53.911791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:33699 (local address 127.6.212.2:58565)
0924 11:33:53.913254 (+ 1463us) negotiation.cc:107] Waiting for socket to connect
0924 11:33:53.913285 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 11:33:53.913443 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:33:53.913849 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:33:53.913855 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:33:53.913883 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:33:53.914322 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:53.914334 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:53.915386 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:53.915391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:53.916919 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:53.916930 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:53.918033 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:53.918039 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:53.918236 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:53.918259 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 11:33:53.918333 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 11:33:53.918469 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1205,"mutex_wait_us":43,"thread_start_us":1378,"threads_started":1}
I20260924 11:33:53.919003 9262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:53.911789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.3:38443 (local address 127.6.212.2:38111)
0924 11:33:53.913409 (+ 1620us) negotiation.cc:107] Waiting for socket to connect
0924 11:33:53.913423 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 11:33:53.913525 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:33:53.913921 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:33:53.913926 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:33:53.913958 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:33:53.914482 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:53.914494 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:53.915515 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:53.915518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:53.916942 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:53.916953 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:53.918068 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:53.918075 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:53.918275 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:53.918297 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 11:33:53.918422 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 11:33:53.918534 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1507,"mutex_wait_us":34,"thread_start_us":1018,"threads_started":1}
I20260924 11:33:53.920116 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29dcb6019dcf4d4fae63a9e378c2203a" candidate_uuid: "67e69286416b452a8de92d69237c5b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c9aac6a3bab408fb9764d14a9b48903" is_pre_election: true
I20260924 11:33:53.920116 9183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29dcb6019dcf4d4fae63a9e378c2203a" candidate_uuid: "67e69286416b452a8de92d69237c5b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" is_pre_election: true
I20260924 11:33:53.920655 9183 raft_consensus.cc:2454] T 29dcb6019dcf4d4fae63a9e378c2203a P 1d9ae86a5c17417c8980ccfc29b037d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67e69286416b452a8de92d69237c5b08 in term 0.
I20260924 11:33:53.920655 8910 raft_consensus.cc:2454] T 29dcb6019dcf4d4fae63a9e378c2203a P 5c9aac6a3bab408fb9764d14a9b48903 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67e69286416b452a8de92d69237c5b08 in term 0.
I20260924 11:33:53.921355 8978 leader_election.cc:302] T 29dcb6019dcf4d4fae63a9e378c2203a P 67e69286416b452a8de92d69237c5b08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1d9ae86a5c17417c8980ccfc29b037d0, 5c9aac6a3bab408fb9764d14a9b48903, 67e69286416b452a8de92d69237c5b08; no voters:
I20260924 11:33:53.921789 9260 raft_consensus.cc:2781] T 29dcb6019dcf4d4fae63a9e378c2203a P 67e69286416b452a8de92d69237c5b08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:33:53.921952 9260 raft_consensus.cc:482] T 29dcb6019dcf4d4fae63a9e378c2203a P 67e69286416b452a8de92d69237c5b08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:33:53.922075 9260 raft_consensus.cc:3037] T 29dcb6019dcf4d4fae63a9e378c2203a P 67e69286416b452a8de92d69237c5b08 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:33:53.924388 9260 raft_consensus.cc:504] T 29dcb6019dcf4d4fae63a9e378c2203a P 67e69286416b452a8de92d69237c5b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } }
I20260924 11:33:53.925050 9260 leader_election.cc:288] T 29dcb6019dcf4d4fae63a9e378c2203a P 67e69286416b452a8de92d69237c5b08 [CANDIDATE]: Term 1 election: Requested vote from peers 5c9aac6a3bab408fb9764d14a9b48903 (127.6.212.1:33699), 1d9ae86a5c17417c8980ccfc29b037d0 (127.6.212.3:38443)
I20260924 11:33:53.925536 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29dcb6019dcf4d4fae63a9e378c2203a" candidate_uuid: "67e69286416b452a8de92d69237c5b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c9aac6a3bab408fb9764d14a9b48903"
I20260924 11:33:53.925628 9183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29dcb6019dcf4d4fae63a9e378c2203a" candidate_uuid: "67e69286416b452a8de92d69237c5b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d9ae86a5c17417c8980ccfc29b037d0"
I20260924 11:33:53.925745 8910 raft_consensus.cc:3037] T 29dcb6019dcf4d4fae63a9e378c2203a P 5c9aac6a3bab408fb9764d14a9b48903 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:33:53.925874 9183 raft_consensus.cc:3037] T 29dcb6019dcf4d4fae63a9e378c2203a P 1d9ae86a5c17417c8980ccfc29b037d0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:33:53.928161 8910 raft_consensus.cc:2454] T 29dcb6019dcf4d4fae63a9e378c2203a P 5c9aac6a3bab408fb9764d14a9b48903 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67e69286416b452a8de92d69237c5b08 in term 1.
I20260924 11:33:53.928319 9183 raft_consensus.cc:2454] T 29dcb6019dcf4d4fae63a9e378c2203a P 1d9ae86a5c17417c8980ccfc29b037d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67e69286416b452a8de92d69237c5b08 in term 1.
I20260924 11:33:53.928602 8978 leader_election.cc:302] T 29dcb6019dcf4d4fae63a9e378c2203a P 67e69286416b452a8de92d69237c5b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c9aac6a3bab408fb9764d14a9b48903, 67e69286416b452a8de92d69237c5b08; no voters:
I20260924 11:33:53.928870 9260 raft_consensus.cc:2781] T 29dcb6019dcf4d4fae63a9e378c2203a P 67e69286416b452a8de92d69237c5b08 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:33:53.929340 9260 raft_consensus.cc:686] T 29dcb6019dcf4d4fae63a9e378c2203a P 67e69286416b452a8de92d69237c5b08 [term 1 LEADER]: Becoming Leader. State: Replica: 67e69286416b452a8de92d69237c5b08, State: Running, Role: LEADER
I20260924 11:33:53.929875 9260 consensus_queue.cc:237] T 29dcb6019dcf4d4fae63a9e378c2203a P 67e69286416b452a8de92d69237c5b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } }
I20260924 11:33:53.934283 8747 catalog_manager.cc:5495] T 29dcb6019dcf4d4fae63a9e378c2203a P 67e69286416b452a8de92d69237c5b08 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67e69286416b452a8de92d69237c5b08 (127.6.212.2). New cstate: current_term: 1 leader_uuid: "67e69286416b452a8de92d69237c5b08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } health_report { overall_health: UNKNOWN } } }
W20260924 11:33:53.945593 9095 tablet.cc:2369] T 29dcb6019dcf4d4fae63a9e378c2203a P 67e69286416b452a8de92d69237c5b08: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:33:53.996248 9234 tablet.cc:2369] T 29dcb6019dcf4d4fae63a9e378c2203a P 1d9ae86a5c17417c8980ccfc29b037d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 11:33:53 dist-test-slave-cq7c krb5kdc[8703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249631, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:33:54.004649 8843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:53.989912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52474 (local address 127.6.212.62:38939)
0924 11:33:53.990075 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:53.990079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:53.990649 (+ 570us) server_negotiation.cc:407] Connection header received
0924 11:33:53.990844 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:53.990849 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:53.990914 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:53.991013 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:33:53.992271 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:53.993110 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:53.994452 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:53.994872 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:53.995012 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:53.998635 (+ 3623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:33:53.998648 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:33:53.998652 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:33:53.998681 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:33:54.001389 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:54.002234 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:54.002249 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:54.002253 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:54.002393 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:54.002824 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:54.002832 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:54.002841 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:54.003139 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:33:54.003482 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:54.003992 (+ 510us) server_negotiation.cc:299] Negotiation successful
0924 11:33:54.004329 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":136,"server-negotiator.queue_time_us":54}
I20260924 11:33:54.004681 9270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:53.989949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:38939 (local address 127.0.0.1:52474)
0924 11:33:53.990546 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0924 11:33:53.990562 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:33:53.990727 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:33:53.991042 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:33:53.991046 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:33:53.991840 (+ 794us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:33:53.992098 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:53.992108 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:53.993239 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:53.993246 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:53.994317 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:53.994329 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:53.995051 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:53.995057 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:53.995220 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:53.995906 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:33:53.995945 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:33:53.998468 (+ 2523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:33:54.001590 (+ 3122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:33:54.001605 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:33:54.001613 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:33:54.002094 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:33:54.002520 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:33:54.002528 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:33:54.002532 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:33:54.002675 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:33:54.003387 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:33:54.003396 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:33:54.003698 (+ 302us) client_negotiation.cc:769] Sending connection context
0924 11:33:54.003903 (+ 205us) client_negotiation.cc:240] Negotiation successful
0924 11:33:54.004193 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":292,"threads_started":1}
I20260924 11:33:54.010074 8747 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:52474:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 11:33:54.011077 8747 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:33:54.018080 8889 tablet_service.cc:1467] Processing CreateTablet for tablet 663d4a678cde45d78ea587ca0afe9b89 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8857a6a019e443608c8f467f6c8d74cd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 11:33:54.018203 9163 tablet_service.cc:1467] Processing CreateTablet for tablet 663d4a678cde45d78ea587ca0afe9b89 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8857a6a019e443608c8f467f6c8d74cd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 11:33:54.019129 8889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 663d4a678cde45d78ea587ca0afe9b89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:33:54.019157 9163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 663d4a678cde45d78ea587ca0afe9b89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:33:54.018914 9024 tablet_service.cc:1467] Processing CreateTablet for tablet 663d4a678cde45d78ea587ca0afe9b89 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8857a6a019e443608c8f467f6c8d74cd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 11:33:54.019718 9024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 663d4a678cde45d78ea587ca0afe9b89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:33:54.024960 9252 tablet_bootstrap.cc:492] T 663d4a678cde45d78ea587ca0afe9b89 P 5c9aac6a3bab408fb9764d14a9b48903: Bootstrap starting.
I20260924 11:33:54.026206 9252 tablet_bootstrap.cc:654] T 663d4a678cde45d78ea587ca0afe9b89 P 5c9aac6a3bab408fb9764d14a9b48903: Neither blocks nor log segments found. Creating new log.
I20260924 11:33:54.026360 9253 tablet_bootstrap.cc:492] T 663d4a678cde45d78ea587ca0afe9b89 P 1d9ae86a5c17417c8980ccfc29b037d0: Bootstrap starting.
I20260924 11:33:54.026389 9255 tablet_bootstrap.cc:492] T 663d4a678cde45d78ea587ca0afe9b89 P 67e69286416b452a8de92d69237c5b08: Bootstrap starting.
I20260924 11:33:54.028162 9253 tablet_bootstrap.cc:654] T 663d4a678cde45d78ea587ca0afe9b89 P 1d9ae86a5c17417c8980ccfc29b037d0: Neither blocks nor log segments found. Creating new log.
I20260924 11:33:54.028187 9255 tablet_bootstrap.cc:654] T 663d4a678cde45d78ea587ca0afe9b89 P 67e69286416b452a8de92d69237c5b08: Neither blocks nor log segments found. Creating new log.
I20260924 11:33:54.028939 9252 tablet_bootstrap.cc:492] T 663d4a678cde45d78ea587ca0afe9b89 P 5c9aac6a3bab408fb9764d14a9b48903: No bootstrap required, opened a new log
I20260924 11:33:54.029125 9252 ts_tablet_manager.cc:1397] T 663d4a678cde45d78ea587ca0afe9b89 P 5c9aac6a3bab408fb9764d14a9b48903: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260924 11:33:54.030030 9252 raft_consensus.cc:348] T 663d4a678cde45d78ea587ca0afe9b89 P 5c9aac6a3bab408fb9764d14a9b48903 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } }
I20260924 11:33:54.030299 9252 raft_consensus.cc:374] T 663d4a678cde45d78ea587ca0afe9b89 P 5c9aac6a3bab408fb9764d14a9b48903 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:33:54.030422 9252 raft_consensus.cc:729] T 663d4a678cde45d78ea587ca0afe9b89 P 5c9aac6a3bab408fb9764d14a9b48903 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c9aac6a3bab408fb9764d14a9b48903, State: Initialized, Role: FOLLOWER
I20260924 11:33:54.030570 9253 tablet_bootstrap.cc:492] T 663d4a678cde45d78ea587ca0afe9b89 P 1d9ae86a5c17417c8980ccfc29b037d0: No bootstrap required, opened a new log
I20260924 11:33:54.030702 9253 ts_tablet_manager.cc:1397] T 663d4a678cde45d78ea587ca0afe9b89 P 1d9ae86a5c17417c8980ccfc29b037d0: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260924 11:33:54.030712 9255 tablet_bootstrap.cc:492] T 663d4a678cde45d78ea587ca0afe9b89 P 67e69286416b452a8de92d69237c5b08: No bootstrap required, opened a new log
I20260924 11:33:54.030663 9252 consensus_queue.cc:260] T 663d4a678cde45d78ea587ca0afe9b89 P 5c9aac6a3bab408fb9764d14a9b48903 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } }
I20260924 11:33:54.030869 9255 ts_tablet_manager.cc:1397] T 663d4a678cde45d78ea587ca0afe9b89 P 67e69286416b452a8de92d69237c5b08: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 11:33:54.031137 9258 tablet_replica.cc:439] T 663d4a678cde45d78ea587ca0afe9b89 P 5c9aac6a3bab408fb9764d14a9b48903: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } } }
I20260924 11:33:54.031551 9258 tablet_replica.cc:442] T 663d4a678cde45d78ea587ca0afe9b89 P 5c9aac6a3bab408fb9764d14a9b48903: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:33:54.031520 9253 raft_consensus.cc:348] T 663d4a678cde45d78ea587ca0afe9b89 P 1d9ae86a5c17417c8980ccfc29b037d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } }
I20260924 11:33:54.031759 9253 raft_consensus.cc:374] T 663d4a678cde45d78ea587ca0afe9b89 P 1d9ae86a5c17417c8980ccfc29b037d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:33:54.031627 9255 raft_consensus.cc:348] T 663d4a678cde45d78ea587ca0afe9b89 P 67e69286416b452a8de92d69237c5b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } }
I20260924 11:33:54.031847 9253 raft_consensus.cc:729] T 663d4a678cde45d78ea587ca0afe9b89 P 1d9ae86a5c17417c8980ccfc29b037d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d9ae86a5c17417c8980ccfc29b037d0, State: Initialized, Role: FOLLOWER
I20260924 11:33:54.032109 9252 ts_tablet_manager.cc:1428] T 663d4a678cde45d78ea587ca0afe9b89 P 5c9aac6a3bab408fb9764d14a9b48903: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 11:33:54.032064 9253 consensus_queue.cc:260] T 663d4a678cde45d78ea587ca0afe9b89 P 1d9ae86a5c17417c8980ccfc29b037d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } }
I20260924 11:33:54.032625 9253 ts_tablet_manager.cc:1428] T 663d4a678cde45d78ea587ca0afe9b89 P 1d9ae86a5c17417c8980ccfc29b037d0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 11:33:54.032723 9255 raft_consensus.cc:374] T 663d4a678cde45d78ea587ca0afe9b89 P 67e69286416b452a8de92d69237c5b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:33:54.032529 9259 tablet_replica.cc:439] T 663d4a678cde45d78ea587ca0afe9b89 P 1d9ae86a5c17417c8980ccfc29b037d0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } } }
I20260924 11:33:54.032857 9255 raft_consensus.cc:729] T 663d4a678cde45d78ea587ca0afe9b89 P 67e69286416b452a8de92d69237c5b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67e69286416b452a8de92d69237c5b08, State: Initialized, Role: FOLLOWER
I20260924 11:33:54.032881 9259 tablet_replica.cc:442] T 663d4a678cde45d78ea587ca0afe9b89 P 1d9ae86a5c17417c8980ccfc29b037d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:33:54.033116 9255 consensus_queue.cc:260] T 663d4a678cde45d78ea587ca0afe9b89 P 67e69286416b452a8de92d69237c5b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } }
I20260924 11:33:54.033574 9260 tablet_replica.cc:439] T 663d4a678cde45d78ea587ca0afe9b89 P 67e69286416b452a8de92d69237c5b08: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } } }
I20260924 11:33:54.033831 9255 ts_tablet_manager.cc:1428] T 663d4a678cde45d78ea587ca0afe9b89 P 67e69286416b452a8de92d69237c5b08: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 11:33:54.033897 9260 tablet_replica.cc:442] T 663d4a678cde45d78ea587ca0afe9b89 P 67e69286416b452a8de92d69237c5b08: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:33:54.036305 9260 raft_consensus.cc:482] T 663d4a678cde45d78ea587ca0afe9b89 P 67e69286416b452a8de92d69237c5b08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:33:54.036480 9260 raft_consensus.cc:504] T 663d4a678cde45d78ea587ca0afe9b89 P 67e69286416b452a8de92d69237c5b08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } }
I20260924 11:33:54.037129 9260 leader_election.cc:288] T 663d4a678cde45d78ea587ca0afe9b89 P 67e69286416b452a8de92d69237c5b08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c9aac6a3bab408fb9764d14a9b48903 (127.6.212.1:33699), 1d9ae86a5c17417c8980ccfc29b037d0 (127.6.212.3:38443)
I20260924 11:33:54.037725 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663d4a678cde45d78ea587ca0afe9b89" candidate_uuid: "67e69286416b452a8de92d69237c5b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c9aac6a3bab408fb9764d14a9b48903" is_pre_election: true
I20260924 11:33:54.037772 9183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663d4a678cde45d78ea587ca0afe9b89" candidate_uuid: "67e69286416b452a8de92d69237c5b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" is_pre_election: true
I20260924 11:33:54.037935 9183 raft_consensus.cc:2454] T 663d4a678cde45d78ea587ca0afe9b89 P 1d9ae86a5c17417c8980ccfc29b037d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67e69286416b452a8de92d69237c5b08 in term 0.
I20260924 11:33:54.037963 8910 raft_consensus.cc:2454] T 663d4a678cde45d78ea587ca0afe9b89 P 5c9aac6a3bab408fb9764d14a9b48903 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67e69286416b452a8de92d69237c5b08 in term 0.
I20260924 11:33:54.038304 8977 leader_election.cc:302] T 663d4a678cde45d78ea587ca0afe9b89 P 67e69286416b452a8de92d69237c5b08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d9ae86a5c17417c8980ccfc29b037d0, 67e69286416b452a8de92d69237c5b08; no voters:
I20260924 11:33:54.038604 9260 raft_consensus.cc:2781] T 663d4a678cde45d78ea587ca0afe9b89 P 67e69286416b452a8de92d69237c5b08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:33:54.038679 9260 raft_consensus.cc:482] T 663d4a678cde45d78ea587ca0afe9b89 P 67e69286416b452a8de92d69237c5b08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:33:54.038797 9260 raft_consensus.cc:3037] T 663d4a678cde45d78ea587ca0afe9b89 P 67e69286416b452a8de92d69237c5b08 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:33:54.040762 9260 raft_consensus.cc:504] T 663d4a678cde45d78ea587ca0afe9b89 P 67e69286416b452a8de92d69237c5b08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } }
I20260924 11:33:54.041267 9260 leader_election.cc:288] T 663d4a678cde45d78ea587ca0afe9b89 P 67e69286416b452a8de92d69237c5b08 [CANDIDATE]: Term 1 election: Requested vote from peers 5c9aac6a3bab408fb9764d14a9b48903 (127.6.212.1:33699), 1d9ae86a5c17417c8980ccfc29b037d0 (127.6.212.3:38443)
I20260924 11:33:54.041641 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663d4a678cde45d78ea587ca0afe9b89" candidate_uuid: "67e69286416b452a8de92d69237c5b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c9aac6a3bab408fb9764d14a9b48903"
I20260924 11:33:54.041657 9183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "663d4a678cde45d78ea587ca0afe9b89" candidate_uuid: "67e69286416b452a8de92d69237c5b08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d9ae86a5c17417c8980ccfc29b037d0"
I20260924 11:33:54.041849 8910 raft_consensus.cc:3037] T 663d4a678cde45d78ea587ca0afe9b89 P 5c9aac6a3bab408fb9764d14a9b48903 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:33:54.041836 9183 raft_consensus.cc:3037] T 663d4a678cde45d78ea587ca0afe9b89 P 1d9ae86a5c17417c8980ccfc29b037d0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:33:54.044092 9183 raft_consensus.cc:2454] T 663d4a678cde45d78ea587ca0afe9b89 P 1d9ae86a5c17417c8980ccfc29b037d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67e69286416b452a8de92d69237c5b08 in term 1.
I20260924 11:33:54.044549 8977 leader_election.cc:302] T 663d4a678cde45d78ea587ca0afe9b89 P 67e69286416b452a8de92d69237c5b08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d9ae86a5c17417c8980ccfc29b037d0, 67e69286416b452a8de92d69237c5b08; no voters:
I20260924 11:33:54.044653 8910 raft_consensus.cc:2454] T 663d4a678cde45d78ea587ca0afe9b89 P 5c9aac6a3bab408fb9764d14a9b48903 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67e69286416b452a8de92d69237c5b08 in term 1.
I20260924 11:33:54.045033 9260 raft_consensus.cc:2781] T 663d4a678cde45d78ea587ca0afe9b89 P 67e69286416b452a8de92d69237c5b08 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:33:54.045199 9260 raft_consensus.cc:686] T 663d4a678cde45d78ea587ca0afe9b89 P 67e69286416b452a8de92d69237c5b08 [term 1 LEADER]: Becoming Leader. State: Replica: 67e69286416b452a8de92d69237c5b08, State: Running, Role: LEADER
I20260924 11:33:54.045429 9260 consensus_queue.cc:237] T 663d4a678cde45d78ea587ca0afe9b89 P 67e69286416b452a8de92d69237c5b08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } }
I20260924 11:33:54.047166 9260 tablet_replica.cc:439] T 663d4a678cde45d78ea587ca0afe9b89 P 67e69286416b452a8de92d69237c5b08: TxnStatusTablet state changed. Reason: New leader 67e69286416b452a8de92d69237c5b08. Latest consensus state: current_term: 1 leader_uuid: "67e69286416b452a8de92d69237c5b08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } } }
I20260924 11:33:54.047529 9260 tablet_replica.cc:442] T 663d4a678cde45d78ea587ca0afe9b89 P 67e69286416b452a8de92d69237c5b08: This TxnStatusTablet replica's current role is: LEADER
I20260924 11:33:54.048141 9275 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 11:33:54.048454 9275 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 11:33:54.048211 8747 catalog_manager.cc:5495] T 663d4a678cde45d78ea587ca0afe9b89 P 67e69286416b452a8de92d69237c5b08 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67e69286416b452a8de92d69237c5b08 (127.6.212.2). New cstate: current_term: 1 leader_uuid: "67e69286416b452a8de92d69237c5b08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } health_report { overall_health: UNKNOWN } } }
I20260924 11:33:54.079007 9247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:54.072790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42934 (local address 127.6.212.2:45989)
0924 11:33:54.072982 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:54.072996 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:54.073022 (+ 26us) server_negotiation.cc:407] Connection header received
0924 11:33:54.073238 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:54.073244 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:54.073362 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:54.073523 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:33:54.073960 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:54.074793 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:54.076641 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:54.077136 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:54.077277 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:54.077840 (+ 563us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:33:54.078369 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:33:54.078544 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:54.078619 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 11:33:54.078697 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 11:33:54.079042 9270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:54.072673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.2:45989 (local address 127.0.0.1:42934)
0924 11:33:54.072878 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 11:33:54.072896 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:33:54.073037 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:33:54.073540 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:33:54.073543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:33:54.073552 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 11:33:54.073811 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:54.073821 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:54.074936 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:54.074943 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:54.076487 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:54.076499 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:54.077310 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:54.077316 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:54.077567 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:54.077630 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 11:33:54.078502 (+ 872us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 11:33:54.078522 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 11:33:54.078595 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 11:33:54.078742 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 11:33:54.086792 9183 raft_consensus.cc:1261] T 663d4a678cde45d78ea587ca0afe9b89 P 1d9ae86a5c17417c8980ccfc29b037d0 [term 1 FOLLOWER]: Refusing update from remote peer 67e69286416b452a8de92d69237c5b08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:33:54.087024 8910 raft_consensus.cc:1261] T 663d4a678cde45d78ea587ca0afe9b89 P 5c9aac6a3bab408fb9764d14a9b48903 [term 1 FOLLOWER]: Refusing update from remote peer 67e69286416b452a8de92d69237c5b08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:33:54.087904 9263 consensus_queue.cc:1035] T 663d4a678cde45d78ea587ca0afe9b89 P 67e69286416b452a8de92d69237c5b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:33:54.088414 9260 consensus_queue.cc:1035] T 663d4a678cde45d78ea587ca0afe9b89 P 67e69286416b452a8de92d69237c5b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:33:54.105407 9259 tablet_replica.cc:439] T 663d4a678cde45d78ea587ca0afe9b89 P 1d9ae86a5c17417c8980ccfc29b037d0: TxnStatusTablet state changed. Reason: New leader 67e69286416b452a8de92d69237c5b08. Latest consensus state: current_term: 1 leader_uuid: "67e69286416b452a8de92d69237c5b08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } } }
I20260924 11:33:54.105713 9259 tablet_replica.cc:442] T 663d4a678cde45d78ea587ca0afe9b89 P 1d9ae86a5c17417c8980ccfc29b037d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:33:54.108340 9258 tablet_replica.cc:439] T 663d4a678cde45d78ea587ca0afe9b89 P 5c9aac6a3bab408fb9764d14a9b48903: TxnStatusTablet state changed. Reason: New leader 67e69286416b452a8de92d69237c5b08. Latest consensus state: current_term: 1 leader_uuid: "67e69286416b452a8de92d69237c5b08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } } }
I20260924 11:33:54.108644 9258 tablet_replica.cc:442] T 663d4a678cde45d78ea587ca0afe9b89 P 5c9aac6a3bab408fb9764d14a9b48903: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:33:54.108763 9278 mvcc.cc:205] Tried to move back new op lower bound from 7332862501209939968 to 7332862501052112896. Current Snapshot: MvccSnapshot[applied={T|T < 7332862501209939968}]
I20260924 11:33:54.110508 9260 tablet_replica.cc:439] T 663d4a678cde45d78ea587ca0afe9b89 P 67e69286416b452a8de92d69237c5b08: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "67e69286416b452a8de92d69237c5b08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } } }
I20260924 11:33:54.110780 9260 tablet_replica.cc:442] T 663d4a678cde45d78ea587ca0afe9b89 P 67e69286416b452a8de92d69237c5b08: This TxnStatusTablet replica's current role is: LEADER
I20260924 11:33:54.112918 9280 mvcc.cc:205] Tried to move back new op lower bound from 7332862501209939968 to 7332862501052112896. Current Snapshot: MvccSnapshot[applied={T|T < 7332862501209939968}]
I20260924 11:33:54.114926 9263 tablet_replica.cc:439] T 663d4a678cde45d78ea587ca0afe9b89 P 67e69286416b452a8de92d69237c5b08: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "67e69286416b452a8de92d69237c5b08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } } }
I20260924 11:33:54.115212 9263 tablet_replica.cc:442] T 663d4a678cde45d78ea587ca0afe9b89 P 67e69286416b452a8de92d69237c5b08: This TxnStatusTablet replica's current role is: LEADER
I20260924 11:33:54.115808 9259 tablet_replica.cc:439] T 663d4a678cde45d78ea587ca0afe9b89 P 1d9ae86a5c17417c8980ccfc29b037d0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "67e69286416b452a8de92d69237c5b08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } } }
I20260924 11:33:54.116020 9259 tablet_replica.cc:442] T 663d4a678cde45d78ea587ca0afe9b89 P 1d9ae86a5c17417c8980ccfc29b037d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:33:54.124822 9258 tablet_replica.cc:439] T 663d4a678cde45d78ea587ca0afe9b89 P 5c9aac6a3bab408fb9764d14a9b48903: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "67e69286416b452a8de92d69237c5b08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } } }
I20260924 11:33:54.125061 9258 tablet_replica.cc:442] T 663d4a678cde45d78ea587ca0afe9b89 P 5c9aac6a3bab408fb9764d14a9b48903: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:33:54.132885 9247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:54.127752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42948 (local address 127.6.212.2:45989)
0924 11:33:54.127912 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:54.127918 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:54.128025 (+ 107us) server_negotiation.cc:407] Connection header received
0924 11:33:54.128252 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:54.128257 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:54.128325 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:54.128464 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:33:54.128822 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:54.129696 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:54.131125 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:54.131625 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:54.131763 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:54.132152 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:33:54.132272 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:33:54.132372 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:54.132511 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 11:33:54.132591 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 11:33:54.150705 9247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:54.144288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:36703 (local address 127.6.212.2:45989)
0924 11:33:54.144501 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:54.144507 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:54.144783 (+ 276us) server_negotiation.cc:407] Connection header received
0924 11:33:54.144946 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:54.144949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:54.145008 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:54.145091 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:33:54.145757 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:54.146628 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:54.148898 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:54.149327 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:54.149422 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:54.150047 (+ 625us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:33:54.150148 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:33:54.150214 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:54.150389 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 11:33:54.150452 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 11:33:54.151031 9289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:54.143996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.2:45989 (local address 127.6.212.2:36703)
0924 11:33:54.144654 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0924 11:33:54.144674 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 11:33:54.144823 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:33:54.145135 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:33:54.145140 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:33:54.145152 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 11:33:54.145587 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:54.145604 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:54.146749 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:54.146754 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:54.148735 (+ 1981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:54.148750 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:54.149509 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:54.149516 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:54.149731 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:54.149805 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 11:33:54.150258 (+ 453us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 11:33:54.150269 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 11:33:54.150366 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 11:33:54.150512 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"mutex_wait_us":4,"thread_start_us":309,"threads_started":1}
I20260924 11:33:54.161116 9183 raft_consensus.cc:1261] T 29dcb6019dcf4d4fae63a9e378c2203a P 1d9ae86a5c17417c8980ccfc29b037d0 [term 1 FOLLOWER]: Refusing update from remote peer 67e69286416b452a8de92d69237c5b08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:33:54.161670 9263 consensus_queue.cc:1035] T 29dcb6019dcf4d4fae63a9e378c2203a P 67e69286416b452a8de92d69237c5b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:33:54.162559 8910 raft_consensus.cc:1261] T 29dcb6019dcf4d4fae63a9e378c2203a P 5c9aac6a3bab408fb9764d14a9b48903 [term 1 FOLLOWER]: Refusing update from remote peer 67e69286416b452a8de92d69237c5b08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:33:54.163852 9263 consensus_queue.cc:1035] T 29dcb6019dcf4d4fae63a9e378c2203a P 67e69286416b452a8de92d69237c5b08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:33:54.172047 9279 mvcc.cc:205] Tried to move back new op lower bound from 7332862501519294464 to 7332862500580843520. Current Snapshot: MvccSnapshot[applied={T|T < 7332862501519294464}]
I20260924 11:33:54.239671 8748 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:33:54.239982 8748 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:33:54.243757 8748 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fa7becf8bf9b4da3b0b57bc5dcadbc18: Sending DeleteTablet for 3 replicas of tablet 29dcb6019dcf4d4fae63a9e378c2203a
I20260924 11:33:54.244793 9024 tablet_service.cc:1514] Processing DeleteTablet for tablet 29dcb6019dcf4d4fae63a9e378c2203a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:33:54 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:42920
I20260924 11:33:54.244975 8889 tablet_service.cc:1514] Processing DeleteTablet for tablet 29dcb6019dcf4d4fae63a9e378c2203a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:33:54 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:53276
I20260924 11:33:54.245008 9163 tablet_service.cc:1514] Processing DeleteTablet for tablet 29dcb6019dcf4d4fae63a9e378c2203a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:33:54 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:51384
I20260924 11:33:54.246850 8748 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52470:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:33:54.247629 8748 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:33:54.253087 9295 tablet_replica.cc:330] T 29dcb6019dcf4d4fae63a9e378c2203a P 5c9aac6a3bab408fb9764d14a9b48903: stopping tablet replica
I20260924 11:33:54.253592 9296 tablet_replica.cc:330] T 29dcb6019dcf4d4fae63a9e378c2203a P 1d9ae86a5c17417c8980ccfc29b037d0: stopping tablet replica
I20260924 11:33:54.254544 9296 raft_consensus.cc:2229] T 29dcb6019dcf4d4fae63a9e378c2203a P 1d9ae86a5c17417c8980ccfc29b037d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:33:54.255223 9295 raft_consensus.cc:2229] T 29dcb6019dcf4d4fae63a9e378c2203a P 5c9aac6a3bab408fb9764d14a9b48903 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:33:54.255970 9296 raft_consensus.cc:2258] T 29dcb6019dcf4d4fae63a9e378c2203a P 1d9ae86a5c17417c8980ccfc29b037d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:33:54.256608 9295 raft_consensus.cc:2258] T 29dcb6019dcf4d4fae63a9e378c2203a P 5c9aac6a3bab408fb9764d14a9b48903 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:33:54.256556 8888 tablet_service.cc:1467] Processing CreateTablet for tablet f4986a2ad3bc42cb846fcdb6030333c7 (DEFAULT_TABLE table=test-table [id=165291ca13ca49a382a116a1117b3049]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:33:54.257161 8888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4986a2ad3bc42cb846fcdb6030333c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:33:54.257560 9023 tablet_service.cc:1467] Processing CreateTablet for tablet f4986a2ad3bc42cb846fcdb6030333c7 (DEFAULT_TABLE table=test-table [id=165291ca13ca49a382a116a1117b3049]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:33:54.258145 9023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4986a2ad3bc42cb846fcdb6030333c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:33:54.259352 9295 ts_tablet_manager.cc:1905] T 29dcb6019dcf4d4fae63a9e378c2203a P 5c9aac6a3bab408fb9764d14a9b48903: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:33:54.264499 9252 tablet_bootstrap.cc:492] T f4986a2ad3bc42cb846fcdb6030333c7 P 5c9aac6a3bab408fb9764d14a9b48903: Bootstrap starting.
I20260924 11:33:54.265431 9294 tablet_replica.cc:330] T 29dcb6019dcf4d4fae63a9e378c2203a P 67e69286416b452a8de92d69237c5b08: stopping tablet replica
I20260924 11:33:54.265998 9294 raft_consensus.cc:2229] T 29dcb6019dcf4d4fae63a9e378c2203a P 67e69286416b452a8de92d69237c5b08 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:33:54.266135 9252 tablet_bootstrap.cc:654] T f4986a2ad3bc42cb846fcdb6030333c7 P 5c9aac6a3bab408fb9764d14a9b48903: Neither blocks nor log segments found. Creating new log.
I20260924 11:33:54.266552 9163 tablet_service.cc:1467] Processing CreateTablet for tablet f4986a2ad3bc42cb846fcdb6030333c7 (DEFAULT_TABLE table=test-table [id=165291ca13ca49a382a116a1117b3049]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:33:54.267037 9255 tablet_bootstrap.cc:492] T f4986a2ad3bc42cb846fcdb6030333c7 P 67e69286416b452a8de92d69237c5b08: Bootstrap starting.
I20260924 11:33:54.267342 9163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4986a2ad3bc42cb846fcdb6030333c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:33:54.268752 9255 tablet_bootstrap.cc:654] T f4986a2ad3bc42cb846fcdb6030333c7 P 67e69286416b452a8de92d69237c5b08: Neither blocks nor log segments found. Creating new log.
I20260924 11:33:54.266701 9294 raft_consensus.cc:2258] T 29dcb6019dcf4d4fae63a9e378c2203a P 67e69286416b452a8de92d69237c5b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:33:54.275188 9296 ts_tablet_manager.cc:1905] T 29dcb6019dcf4d4fae63a9e378c2203a P 1d9ae86a5c17417c8980ccfc29b037d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:33:54.276572 9255 tablet_bootstrap.cc:492] T f4986a2ad3bc42cb846fcdb6030333c7 P 67e69286416b452a8de92d69237c5b08: No bootstrap required, opened a new log
I20260924 11:33:54.276822 9255 ts_tablet_manager.cc:1397] T f4986a2ad3bc42cb846fcdb6030333c7 P 67e69286416b452a8de92d69237c5b08: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20260924 11:33:54.277710 9294 ts_tablet_manager.cc:1905] T 29dcb6019dcf4d4fae63a9e378c2203a P 67e69286416b452a8de92d69237c5b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:33:54.277614 9255 raft_consensus.cc:348] T f4986a2ad3bc42cb846fcdb6030333c7 P 67e69286416b452a8de92d69237c5b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } }
I20260924 11:33:54.278076 9255 raft_consensus.cc:374] T f4986a2ad3bc42cb846fcdb6030333c7 P 67e69286416b452a8de92d69237c5b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:33:54.278193 9255 raft_consensus.cc:729] T f4986a2ad3bc42cb846fcdb6030333c7 P 67e69286416b452a8de92d69237c5b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67e69286416b452a8de92d69237c5b08, State: Initialized, Role: FOLLOWER
I20260924 11:33:54.278510 9255 consensus_queue.cc:260] T f4986a2ad3bc42cb846fcdb6030333c7 P 67e69286416b452a8de92d69237c5b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } }
I20260924 11:33:54.279385 9253 tablet_bootstrap.cc:492] T f4986a2ad3bc42cb846fcdb6030333c7 P 1d9ae86a5c17417c8980ccfc29b037d0: Bootstrap starting.
I20260924 11:33:54.281011 9253 tablet_bootstrap.cc:654] T f4986a2ad3bc42cb846fcdb6030333c7 P 1d9ae86a5c17417c8980ccfc29b037d0: Neither blocks nor log segments found. Creating new log.
I20260924 11:33:54.281298 9255 ts_tablet_manager.cc:1428] T f4986a2ad3bc42cb846fcdb6030333c7 P 67e69286416b452a8de92d69237c5b08: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260924 11:33:54.288146 9253 tablet_bootstrap.cc:492] T f4986a2ad3bc42cb846fcdb6030333c7 P 1d9ae86a5c17417c8980ccfc29b037d0: No bootstrap required, opened a new log
I20260924 11:33:54.288335 9253 ts_tablet_manager.cc:1397] T f4986a2ad3bc42cb846fcdb6030333c7 P 1d9ae86a5c17417c8980ccfc29b037d0: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260924 11:33:54.288686 9295 ts_tablet_manager.cc:1918] T 29dcb6019dcf4d4fae63a9e378c2203a P 5c9aac6a3bab408fb9764d14a9b48903: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 11:33:54.288848 9295 log.cc:1196] T 29dcb6019dcf4d4fae63a9e378c2203a P 5c9aac6a3bab408fb9764d14a9b48903: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-0/wal/wals/29dcb6019dcf4d4fae63a9e378c2203a
I20260924 11:33:54.289149 9253 raft_consensus.cc:348] T f4986a2ad3bc42cb846fcdb6030333c7 P 1d9ae86a5c17417c8980ccfc29b037d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } }
I20260924 11:33:54.289364 9295 ts_tablet_manager.cc:1939] T 29dcb6019dcf4d4fae63a9e378c2203a P 5c9aac6a3bab408fb9764d14a9b48903: Deleting consensus metadata
I20260924 11:33:54.289420 9253 raft_consensus.cc:374] T f4986a2ad3bc42cb846fcdb6030333c7 P 1d9ae86a5c17417c8980ccfc29b037d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:33:54.289491 9253 raft_consensus.cc:729] T f4986a2ad3bc42cb846fcdb6030333c7 P 1d9ae86a5c17417c8980ccfc29b037d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d9ae86a5c17417c8980ccfc29b037d0, State: Initialized, Role: FOLLOWER
I20260924 11:33:54.289775 9253 consensus_queue.cc:260] T f4986a2ad3bc42cb846fcdb6030333c7 P 1d9ae86a5c17417c8980ccfc29b037d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } }
I20260924 11:33:54.291183 8735 catalog_manager.cc:4841] TS 5c9aac6a3bab408fb9764d14a9b48903 (127.6.212.1:33699): tablet 29dcb6019dcf4d4fae63a9e378c2203a (table test-table [id=092c5a403f544dbaaf79ad874d5dff2e]) successfully deleted
I20260924 11:33:54.292218 9253 ts_tablet_manager.cc:1428] T f4986a2ad3bc42cb846fcdb6030333c7 P 1d9ae86a5c17417c8980ccfc29b037d0: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260924 11:33:54.292237 9296 ts_tablet_manager.cc:1918] T 29dcb6019dcf4d4fae63a9e378c2203a P 1d9ae86a5c17417c8980ccfc29b037d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 11:33:54.292621 9296 log.cc:1196] T 29dcb6019dcf4d4fae63a9e378c2203a P 1d9ae86a5c17417c8980ccfc29b037d0: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-2/wal/wals/29dcb6019dcf4d4fae63a9e378c2203a
I20260924 11:33:54.293138 9296 ts_tablet_manager.cc:1939] T 29dcb6019dcf4d4fae63a9e378c2203a P 1d9ae86a5c17417c8980ccfc29b037d0: Deleting consensus metadata
I20260924 11:33:54.294806 8734 catalog_manager.cc:4841] TS 1d9ae86a5c17417c8980ccfc29b037d0 (127.6.212.3:38443): tablet 29dcb6019dcf4d4fae63a9e378c2203a (table test-table [id=092c5a403f544dbaaf79ad874d5dff2e]) successfully deleted
I20260924 11:33:54.296165 9252 tablet_bootstrap.cc:492] T f4986a2ad3bc42cb846fcdb6030333c7 P 5c9aac6a3bab408fb9764d14a9b48903: No bootstrap required, opened a new log
I20260924 11:33:54.296329 9252 ts_tablet_manager.cc:1397] T f4986a2ad3bc42cb846fcdb6030333c7 P 5c9aac6a3bab408fb9764d14a9b48903: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.007s
I20260924 11:33:54.296597 9294 ts_tablet_manager.cc:1918] T 29dcb6019dcf4d4fae63a9e378c2203a P 67e69286416b452a8de92d69237c5b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 11:33:54.296761 9294 log.cc:1196] T 29dcb6019dcf4d4fae63a9e378c2203a P 67e69286416b452a8de92d69237c5b08: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-1/wal/wals/29dcb6019dcf4d4fae63a9e378c2203a
I20260924 11:33:54.297039 9252 raft_consensus.cc:348] T f4986a2ad3bc42cb846fcdb6030333c7 P 5c9aac6a3bab408fb9764d14a9b48903 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } }
I20260924 11:33:54.297248 9294 ts_tablet_manager.cc:1939] T 29dcb6019dcf4d4fae63a9e378c2203a P 67e69286416b452a8de92d69237c5b08: Deleting consensus metadata
I20260924 11:33:54.297310 9252 raft_consensus.cc:374] T f4986a2ad3bc42cb846fcdb6030333c7 P 5c9aac6a3bab408fb9764d14a9b48903 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:33:54.297391 9252 raft_consensus.cc:729] T f4986a2ad3bc42cb846fcdb6030333c7 P 5c9aac6a3bab408fb9764d14a9b48903 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c9aac6a3bab408fb9764d14a9b48903, State: Initialized, Role: FOLLOWER
I20260924 11:33:54.297650 9252 consensus_queue.cc:260] T f4986a2ad3bc42cb846fcdb6030333c7 P 5c9aac6a3bab408fb9764d14a9b48903 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } }
I20260924 11:33:54.298235 9252 ts_tablet_manager.cc:1428] T f4986a2ad3bc42cb846fcdb6030333c7 P 5c9aac6a3bab408fb9764d14a9b48903: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260924 11:33:54.298816 8733 catalog_manager.cc:4841] TS 67e69286416b452a8de92d69237c5b08 (127.6.212.2:45989): tablet 29dcb6019dcf4d4fae63a9e378c2203a (table test-table [id=092c5a403f544dbaaf79ad874d5dff2e]) successfully deleted
I20260924 11:33:54.497318 9258 raft_consensus.cc:482] T f4986a2ad3bc42cb846fcdb6030333c7 P 5c9aac6a3bab408fb9764d14a9b48903 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:33:54.497584 9258 raft_consensus.cc:504] T f4986a2ad3bc42cb846fcdb6030333c7 P 5c9aac6a3bab408fb9764d14a9b48903 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } }
I20260924 11:33:54.499236 9258 leader_election.cc:288] T f4986a2ad3bc42cb846fcdb6030333c7 P 5c9aac6a3bab408fb9764d14a9b48903 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67e69286416b452a8de92d69237c5b08 (127.6.212.2:45989), 1d9ae86a5c17417c8980ccfc29b037d0 (127.6.212.3:38443)
I20260924 11:33:54.507797 9247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:54.499617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:59871 (local address 127.6.212.2:45989)
0924 11:33:54.499901 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:54.499915 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:54.501154 (+ 1239us) server_negotiation.cc:407] Connection header received
0924 11:33:54.501327 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:54.501333 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:54.501408 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:54.501487 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:33:54.502191 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:54.503242 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:54.505623 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:54.506447 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:54.506607 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:54.506705 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:54.507328 (+ 623us) server_negotiation.cc:299] Negotiation successful
0924 11:33:54.507484 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 11:33:54.507884 9301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:54.499692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:48569 (local address 127.6.212.3:38443)
0924 11:33:54.500318 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:54.500326 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:54.500852 (+ 526us) server_negotiation.cc:407] Connection header received
0924 11:33:54.501120 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:54.501126 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:54.501208 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:54.501363 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:33:54.501990 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:54.502902 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:54.505451 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:54.506645 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:54.506779 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:54.506875 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:54.507434 (+ 559us) server_negotiation.cc:299] Negotiation successful
0924 11:33:54.507575 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":380,"threads_started":1}
I20260924 11:33:54.508673 9302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:54.499536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.3:38443 (local address 127.6.212.1:48569)
0924 11:33:54.500720 (+ 1184us) negotiation.cc:107] Waiting for socket to connect
0924 11:33:54.500755 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 11:33:54.500966 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:33:54.501406 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:33:54.501411 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:33:54.501436 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:33:54.501726 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:54.501738 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:54.503034 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:54.503040 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:54.505274 (+ 2234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:54.505289 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:54.507027 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:54.507033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:54.507210 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:54.507241 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 11:33:54.508053 (+ 812us) client_negotiation.cc:240] Negotiation successful
0924 11:33:54.508287 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1037,"mutex_wait_us":20,"thread_start_us":1207,"threads_started":1}
I20260924 11:33:54.508793 9300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:54.499984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.2:45989 (local address 127.6.212.1:59871)
0924 11:33:54.501048 (+ 1064us) negotiation.cc:107] Waiting for socket to connect
0924 11:33:54.501065 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:33:54.501195 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:33:54.501556 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:33:54.501560 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:33:54.501585 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:33:54.502024 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:54.502037 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:54.503409 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:54.503414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:54.505485 (+ 2071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:54.505494 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:54.506941 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:54.506946 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:54.507126 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:54.507158 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 11:33:54.508328 (+ 1170us) client_negotiation.cc:240] Negotiation successful
0924 11:33:54.508432 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":924,"thread_start_us":456,"threads_started":1}
I20260924 11:33:54.509462 9183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4986a2ad3bc42cb846fcdb6030333c7" candidate_uuid: "5c9aac6a3bab408fb9764d14a9b48903" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" is_pre_election: true
I20260924 11:33:54.509713 9183 raft_consensus.cc:2454] T f4986a2ad3bc42cb846fcdb6030333c7 P 1d9ae86a5c17417c8980ccfc29b037d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c9aac6a3bab408fb9764d14a9b48903 in term 0.
I20260924 11:33:54.509635 9046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4986a2ad3bc42cb846fcdb6030333c7" candidate_uuid: "5c9aac6a3bab408fb9764d14a9b48903" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e69286416b452a8de92d69237c5b08" is_pre_election: true
I20260924 11:33:54.510046 9046 raft_consensus.cc:2454] T f4986a2ad3bc42cb846fcdb6030333c7 P 67e69286416b452a8de92d69237c5b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c9aac6a3bab408fb9764d14a9b48903 in term 0.
I20260924 11:33:54.510219 8837 leader_election.cc:302] T f4986a2ad3bc42cb846fcdb6030333c7 P 5c9aac6a3bab408fb9764d14a9b48903 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d9ae86a5c17417c8980ccfc29b037d0, 5c9aac6a3bab408fb9764d14a9b48903; no voters:
I20260924 11:33:54.510671 9258 raft_consensus.cc:2781] T f4986a2ad3bc42cb846fcdb6030333c7 P 5c9aac6a3bab408fb9764d14a9b48903 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:33:54.510828 9258 raft_consensus.cc:482] T f4986a2ad3bc42cb846fcdb6030333c7 P 5c9aac6a3bab408fb9764d14a9b48903 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:33:54.510928 9258 raft_consensus.cc:3037] T f4986a2ad3bc42cb846fcdb6030333c7 P 5c9aac6a3bab408fb9764d14a9b48903 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:33:54.514017 9258 raft_consensus.cc:504] T f4986a2ad3bc42cb846fcdb6030333c7 P 5c9aac6a3bab408fb9764d14a9b48903 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } }
I20260924 11:33:54.514695 9258 leader_election.cc:288] T f4986a2ad3bc42cb846fcdb6030333c7 P 5c9aac6a3bab408fb9764d14a9b48903 [CANDIDATE]: Term 1 election: Requested vote from peers 67e69286416b452a8de92d69237c5b08 (127.6.212.2:45989), 1d9ae86a5c17417c8980ccfc29b037d0 (127.6.212.3:38443)
I20260924 11:33:54.515204 9046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4986a2ad3bc42cb846fcdb6030333c7" candidate_uuid: "5c9aac6a3bab408fb9764d14a9b48903" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e69286416b452a8de92d69237c5b08"
I20260924 11:33:54.515194 9183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4986a2ad3bc42cb846fcdb6030333c7" candidate_uuid: "5c9aac6a3bab408fb9764d14a9b48903" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d9ae86a5c17417c8980ccfc29b037d0"
I20260924 11:33:54.515408 9183 raft_consensus.cc:3037] T f4986a2ad3bc42cb846fcdb6030333c7 P 1d9ae86a5c17417c8980ccfc29b037d0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:33:54.515436 9046 raft_consensus.cc:3037] T f4986a2ad3bc42cb846fcdb6030333c7 P 67e69286416b452a8de92d69237c5b08 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:33:54.517603 9046 raft_consensus.cc:2454] T f4986a2ad3bc42cb846fcdb6030333c7 P 67e69286416b452a8de92d69237c5b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c9aac6a3bab408fb9764d14a9b48903 in term 1.
I20260924 11:33:54.518074 8836 leader_election.cc:302] T f4986a2ad3bc42cb846fcdb6030333c7 P 5c9aac6a3bab408fb9764d14a9b48903 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c9aac6a3bab408fb9764d14a9b48903, 67e69286416b452a8de92d69237c5b08; no voters:
I20260924 11:33:54.518356 9258 raft_consensus.cc:2781] T f4986a2ad3bc42cb846fcdb6030333c7 P 5c9aac6a3bab408fb9764d14a9b48903 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:33:54.518415 9183 raft_consensus.cc:2454] T f4986a2ad3bc42cb846fcdb6030333c7 P 1d9ae86a5c17417c8980ccfc29b037d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c9aac6a3bab408fb9764d14a9b48903 in term 1.
I20260924 11:33:54.519047 9258 raft_consensus.cc:686] T f4986a2ad3bc42cb846fcdb6030333c7 P 5c9aac6a3bab408fb9764d14a9b48903 [term 1 LEADER]: Becoming Leader. State: Replica: 5c9aac6a3bab408fb9764d14a9b48903, State: Running, Role: LEADER
I20260924 11:33:54.519686 9258 consensus_queue.cc:237] T f4986a2ad3bc42cb846fcdb6030333c7 P 5c9aac6a3bab408fb9764d14a9b48903 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } }
I20260924 11:33:54.523285 8748 catalog_manager.cc:5495] T f4986a2ad3bc42cb846fcdb6030333c7 P 5c9aac6a3bab408fb9764d14a9b48903 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c9aac6a3bab408fb9764d14a9b48903 (127.6.212.1). New cstate: current_term: 1 leader_uuid: "5c9aac6a3bab408fb9764d14a9b48903" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } health_report { overall_health: UNKNOWN } } }
I20260924 11:33:54.592532 9305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:54.578149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53286 (local address 127.6.212.1:33699)
0924 11:33:54.586742 (+ 8593us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:54.586748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:54.586802 (+ 54us) server_negotiation.cc:407] Connection header received
0924 11:33:54.586922 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:54.586926 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:54.587016 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:54.587219 (+ 203us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:33:54.587751 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:54.588590 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:54.590064 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:54.590488 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:54.590623 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:54.591088 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:33:54.591805 (+ 717us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:33:54.591965 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:54.592061 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 11:33:54.592139 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8458,"thread_start_us":8349,"threads_started":1}
I20260924 11:33:54.613210 9308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:54.602857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.2:45989 (local address 127.6.212.1:46463)
0924 11:33:54.607901 (+ 5044us) negotiation.cc:107] Waiting for socket to connect
0924 11:33:54.607917 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:33:54.608083 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:33:54.608418 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:33:54.608423 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:33:54.608435 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 11:33:54.608780 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:54.608788 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:54.609911 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:54.609916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:54.610948 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:54.610959 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:54.611986 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:54.611991 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:54.612111 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:54.612165 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 11:33:54.612658 (+ 493us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 11:33:54.612667 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 11:33:54.612728 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 11:33:54.612904 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4899,"thread_start_us":4732,"threads_started":1}
I20260924 11:33:54.613200 9247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:54.603060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:46463 (local address 127.6.212.2:45989)
0924 11:33:54.603311 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:54.603320 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:54.608010 (+ 4690us) server_negotiation.cc:407] Connection header received
0924 11:33:54.608232 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:54.608236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:54.608299 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:54.608454 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:33:54.608959 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:54.609775 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:54.611102 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:54.611704 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:54.611863 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:54.612398 (+ 535us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:33:54.612555 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:33:54.612653 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:54.612797 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 11:33:54.612907 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 11:33:54.623205 9183 raft_consensus.cc:1261] T f4986a2ad3bc42cb846fcdb6030333c7 P 1d9ae86a5c17417c8980ccfc29b037d0 [term 1 FOLLOWER]: Refusing update from remote peer 5c9aac6a3bab408fb9764d14a9b48903: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:33:54.623628 9046 raft_consensus.cc:1261] T f4986a2ad3bc42cb846fcdb6030333c7 P 67e69286416b452a8de92d69237c5b08 [term 1 FOLLOWER]: Refusing update from remote peer 5c9aac6a3bab408fb9764d14a9b48903: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:33:54.624145 9303 consensus_queue.cc:1035] T f4986a2ad3bc42cb846fcdb6030333c7 P 5c9aac6a3bab408fb9764d14a9b48903 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:33:54.624490 9258 consensus_queue.cc:1035] T f4986a2ad3bc42cb846fcdb6030333c7 P 5c9aac6a3bab408fb9764d14a9b48903 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:33:54.655616 9289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:54.649397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:33699 (local address 127.6.212.2:48415)
0924 11:33:54.649553 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 11:33:54.649577 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 11:33:54.649666 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:33:54.650118 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:33:54.650121 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:33:54.650129 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 11:33:54.650278 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:54.650285 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:54.651666 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:54.651672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:54.653355 (+ 1683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:54.653373 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:54.654421 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:54.654427 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:54.654625 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:54.654673 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 11:33:54.655120 (+ 447us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 11:33:54.655130 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 11:33:54.655198 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 11:33:54.655300 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 11:33:54.655629 9305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:54.649659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:48415 (local address 127.6.212.1:33699)
0924 11:33:54.649826 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:54.649832 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:54.649851 (+ 19us) server_negotiation.cc:407] Connection header received
0924 11:33:54.649916 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:54.649922 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:54.650008 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:54.650102 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:33:54.650421 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:54.651524 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:54.653536 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:54.654191 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:54.654310 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:54.654892 (+ 582us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:33:54.655027 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:33:54.655117 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:54.655220 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 11:33:54.655299 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260924 11:33:54.656833 9279 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 11:33:56.270771 8748 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:33:56.271014 8748 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:33:56.275530 8748 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fa7becf8bf9b4da3b0b57bc5dcadbc18: Sending DeleteTablet for 3 replicas of tablet f4986a2ad3bc42cb846fcdb6030333c7
I20260924 11:33:56.276289 8888 tablet_service.cc:1514] Processing DeleteTablet for tablet f4986a2ad3bc42cb846fcdb6030333c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:33:56 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:53276
I20260924 11:33:56.276490 9023 tablet_service.cc:1514] Processing DeleteTablet for tablet f4986a2ad3bc42cb846fcdb6030333c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:33:56 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:42920
I20260924 11:33:56.276678 9163 tablet_service.cc:1514] Processing DeleteTablet for tablet f4986a2ad3bc42cb846fcdb6030333c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:33:56 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:51384
I20260924 11:33:56.277129 9335 tablet_replica.cc:330] T f4986a2ad3bc42cb846fcdb6030333c7 P 5c9aac6a3bab408fb9764d14a9b48903: stopping tablet replica
I20260924 11:33:56.277312 9335 raft_consensus.cc:2229] T f4986a2ad3bc42cb846fcdb6030333c7 P 5c9aac6a3bab408fb9764d14a9b48903 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:33:56.277581 9337 tablet_replica.cc:330] T f4986a2ad3bc42cb846fcdb6030333c7 P 1d9ae86a5c17417c8980ccfc29b037d0: stopping tablet replica
I20260924 11:33:56.278000 9336 tablet_replica.cc:330] T f4986a2ad3bc42cb846fcdb6030333c7 P 67e69286416b452a8de92d69237c5b08: stopping tablet replica
I20260924 11:33:56.278215 9336 raft_consensus.cc:2229] T f4986a2ad3bc42cb846fcdb6030333c7 P 67e69286416b452a8de92d69237c5b08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:33:56.278478 9335 raft_consensus.cc:2258] T f4986a2ad3bc42cb846fcdb6030333c7 P 5c9aac6a3bab408fb9764d14a9b48903 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:33:56.278467 9337 raft_consensus.cc:2229] T f4986a2ad3bc42cb846fcdb6030333c7 P 1d9ae86a5c17417c8980ccfc29b037d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:33:56.278534 9336 raft_consensus.cc:2258] T f4986a2ad3bc42cb846fcdb6030333c7 P 67e69286416b452a8de92d69237c5b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:33:56.278761 9337 raft_consensus.cc:2258] T f4986a2ad3bc42cb846fcdb6030333c7 P 1d9ae86a5c17417c8980ccfc29b037d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:33:56.279600 9336 ts_tablet_manager.cc:1905] T f4986a2ad3bc42cb846fcdb6030333c7 P 67e69286416b452a8de92d69237c5b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:33:56.279631 9335 ts_tablet_manager.cc:1905] T f4986a2ad3bc42cb846fcdb6030333c7 P 5c9aac6a3bab408fb9764d14a9b48903: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:33:56.279808 9337 ts_tablet_manager.cc:1905] T f4986a2ad3bc42cb846fcdb6030333c7 P 1d9ae86a5c17417c8980ccfc29b037d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:33:56.286404 9336 ts_tablet_manager.cc:1918] T f4986a2ad3bc42cb846fcdb6030333c7 P 67e69286416b452a8de92d69237c5b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 11:33:56.286404 9337 ts_tablet_manager.cc:1918] T f4986a2ad3bc42cb846fcdb6030333c7 P 1d9ae86a5c17417c8980ccfc29b037d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 11:33:56.286538 9337 log.cc:1196] T f4986a2ad3bc42cb846fcdb6030333c7 P 1d9ae86a5c17417c8980ccfc29b037d0: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-2/wal/wals/f4986a2ad3bc42cb846fcdb6030333c7
I20260924 11:33:56.286602 9336 log.cc:1196] T f4986a2ad3bc42cb846fcdb6030333c7 P 67e69286416b452a8de92d69237c5b08: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-1/wal/wals/f4986a2ad3bc42cb846fcdb6030333c7
I20260924 11:33:56.287060 9337 ts_tablet_manager.cc:1939] T f4986a2ad3bc42cb846fcdb6030333c7 P 1d9ae86a5c17417c8980ccfc29b037d0: Deleting consensus metadata
I20260924 11:33:56.287142 9336 ts_tablet_manager.cc:1939] T f4986a2ad3bc42cb846fcdb6030333c7 P 67e69286416b452a8de92d69237c5b08: Deleting consensus metadata
I20260924 11:33:56.287352 9335 ts_tablet_manager.cc:1918] T f4986a2ad3bc42cb846fcdb6030333c7 P 5c9aac6a3bab408fb9764d14a9b48903: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 11:33:56.287606 9335 log.cc:1196] T f4986a2ad3bc42cb846fcdb6030333c7 P 5c9aac6a3bab408fb9764d14a9b48903: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-0/wal/wals/f4986a2ad3bc42cb846fcdb6030333c7
I20260924 11:33:56.288134 9335 ts_tablet_manager.cc:1939] T f4986a2ad3bc42cb846fcdb6030333c7 P 5c9aac6a3bab408fb9764d14a9b48903: Deleting consensus metadata
I20260924 11:33:56.288257 8734 catalog_manager.cc:4841] TS 1d9ae86a5c17417c8980ccfc29b037d0 (127.6.212.3:38443): tablet f4986a2ad3bc42cb846fcdb6030333c7 (table test-table [id=165291ca13ca49a382a116a1117b3049]) successfully deleted
I20260924 11:33:56.288304 8733 catalog_manager.cc:4841] TS 67e69286416b452a8de92d69237c5b08 (127.6.212.2:45989): tablet f4986a2ad3bc42cb846fcdb6030333c7 (table test-table [id=165291ca13ca49a382a116a1117b3049]) successfully deleted
I20260924 11:33:56.289285 8735 catalog_manager.cc:4841] TS 5c9aac6a3bab408fb9764d14a9b48903 (127.6.212.1:33699): tablet f4986a2ad3bc42cb846fcdb6030333c7 (table test-table [id=165291ca13ca49a382a116a1117b3049]) successfully deleted
Sep 24 11:33:56 dist-test-slave-cq7c krb5kdc[8703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249636, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 11:33:56 dist-test-slave-cq7c krb5kdc[8703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249636, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:33:56.327864 9345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:56.314047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48036 (local address 127.6.212.62:38939)
0924 11:33:56.314621 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:56.314630 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:56.314710 (+ 80us) server_negotiation.cc:407] Connection header received
0924 11:33:56.314966 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:56.314970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:56.315031 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:56.315143 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:33:56.316463 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:56.317185 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:56.318476 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:56.318872 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:56.319005 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:56.322801 (+ 3796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:33:56.322806 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:33:56.322810 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:33:56.322841 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:33:56.325283 (+ 2442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:56.325891 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:56.325898 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:56.325902 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:56.326024 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:56.326375 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:56.326381 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:56.326384 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:56.326694 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:33:56.326969 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:56.327248 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 11:33:56.327504 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":288,"threads_started":1}
I20260924 11:33:56.331877 8748 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48036:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:33:56.332549 8748 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:33:56.340026 8888 tablet_service.cc:1467] Processing CreateTablet for tablet cde220d6c53e4bb99a53fd3f00f05e2c (DEFAULT_TABLE table=test-table [id=9ff98c7ee0904aa895111e0d21cbe2c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:33:56.340329 9023 tablet_service.cc:1467] Processing CreateTablet for tablet cde220d6c53e4bb99a53fd3f00f05e2c (DEFAULT_TABLE table=test-table [id=9ff98c7ee0904aa895111e0d21cbe2c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:33:56.340364 9163 tablet_service.cc:1467] Processing CreateTablet for tablet cde220d6c53e4bb99a53fd3f00f05e2c (DEFAULT_TABLE table=test-table [id=9ff98c7ee0904aa895111e0d21cbe2c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:33:56.340544 8888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cde220d6c53e4bb99a53fd3f00f05e2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:33:56.340791 9163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cde220d6c53e4bb99a53fd3f00f05e2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:33:56.340794 9023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cde220d6c53e4bb99a53fd3f00f05e2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:33:56.345837 9346 tablet_bootstrap.cc:492] T cde220d6c53e4bb99a53fd3f00f05e2c P 5c9aac6a3bab408fb9764d14a9b48903: Bootstrap starting.
I20260924 11:33:56.347522 9346 tablet_bootstrap.cc:654] T cde220d6c53e4bb99a53fd3f00f05e2c P 5c9aac6a3bab408fb9764d14a9b48903: Neither blocks nor log segments found. Creating new log.
I20260924 11:33:56.348484 9347 tablet_bootstrap.cc:492] T cde220d6c53e4bb99a53fd3f00f05e2c P 1d9ae86a5c17417c8980ccfc29b037d0: Bootstrap starting.
I20260924 11:33:56.350503 9346 tablet_bootstrap.cc:492] T cde220d6c53e4bb99a53fd3f00f05e2c P 5c9aac6a3bab408fb9764d14a9b48903: No bootstrap required, opened a new log
I20260924 11:33:56.350512 9347 tablet_bootstrap.cc:654] T cde220d6c53e4bb99a53fd3f00f05e2c P 1d9ae86a5c17417c8980ccfc29b037d0: Neither blocks nor log segments found. Creating new log.
I20260924 11:33:56.350639 9346 ts_tablet_manager.cc:1397] T cde220d6c53e4bb99a53fd3f00f05e2c P 5c9aac6a3bab408fb9764d14a9b48903: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260924 11:33:56.351562 9346 raft_consensus.cc:348] T cde220d6c53e4bb99a53fd3f00f05e2c P 5c9aac6a3bab408fb9764d14a9b48903 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } }
I20260924 11:33:56.351825 9346 raft_consensus.cc:374] T cde220d6c53e4bb99a53fd3f00f05e2c P 5c9aac6a3bab408fb9764d14a9b48903 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:33:56.351919 9346 raft_consensus.cc:729] T cde220d6c53e4bb99a53fd3f00f05e2c P 5c9aac6a3bab408fb9764d14a9b48903 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c9aac6a3bab408fb9764d14a9b48903, State: Initialized, Role: FOLLOWER
I20260924 11:33:56.352129 9348 tablet_bootstrap.cc:492] T cde220d6c53e4bb99a53fd3f00f05e2c P 67e69286416b452a8de92d69237c5b08: Bootstrap starting.
I20260924 11:33:56.352181 9346 consensus_queue.cc:260] T cde220d6c53e4bb99a53fd3f00f05e2c P 5c9aac6a3bab408fb9764d14a9b48903 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } }
I20260924 11:33:56.353220 9347 tablet_bootstrap.cc:492] T cde220d6c53e4bb99a53fd3f00f05e2c P 1d9ae86a5c17417c8980ccfc29b037d0: No bootstrap required, opened a new log
I20260924 11:33:56.353591 9347 ts_tablet_manager.cc:1397] T cde220d6c53e4bb99a53fd3f00f05e2c P 1d9ae86a5c17417c8980ccfc29b037d0: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260924 11:33:56.354166 9348 tablet_bootstrap.cc:654] T cde220d6c53e4bb99a53fd3f00f05e2c P 67e69286416b452a8de92d69237c5b08: Neither blocks nor log segments found. Creating new log.
I20260924 11:33:56.354679 9346 ts_tablet_manager.cc:1428] T cde220d6c53e4bb99a53fd3f00f05e2c P 5c9aac6a3bab408fb9764d14a9b48903: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260924 11:33:56.354583 9347 raft_consensus.cc:348] T cde220d6c53e4bb99a53fd3f00f05e2c P 1d9ae86a5c17417c8980ccfc29b037d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } }
I20260924 11:33:56.354993 9347 raft_consensus.cc:374] T cde220d6c53e4bb99a53fd3f00f05e2c P 1d9ae86a5c17417c8980ccfc29b037d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:33:56.355188 9347 raft_consensus.cc:729] T cde220d6c53e4bb99a53fd3f00f05e2c P 1d9ae86a5c17417c8980ccfc29b037d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d9ae86a5c17417c8980ccfc29b037d0, State: Initialized, Role: FOLLOWER
I20260924 11:33:56.355526 9347 consensus_queue.cc:260] T cde220d6c53e4bb99a53fd3f00f05e2c P 1d9ae86a5c17417c8980ccfc29b037d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } }
I20260924 11:33:56.356420 9348 tablet_bootstrap.cc:492] T cde220d6c53e4bb99a53fd3f00f05e2c P 67e69286416b452a8de92d69237c5b08: No bootstrap required, opened a new log
I20260924 11:33:56.356419 9347 ts_tablet_manager.cc:1428] T cde220d6c53e4bb99a53fd3f00f05e2c P 1d9ae86a5c17417c8980ccfc29b037d0: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260924 11:33:56.356611 9348 ts_tablet_manager.cc:1397] T cde220d6c53e4bb99a53fd3f00f05e2c P 67e69286416b452a8de92d69237c5b08: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 11:33:56.357398 9348 raft_consensus.cc:348] T cde220d6c53e4bb99a53fd3f00f05e2c P 67e69286416b452a8de92d69237c5b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } }
I20260924 11:33:56.357580 9348 raft_consensus.cc:374] T cde220d6c53e4bb99a53fd3f00f05e2c P 67e69286416b452a8de92d69237c5b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:33:56.357688 9348 raft_consensus.cc:729] T cde220d6c53e4bb99a53fd3f00f05e2c P 67e69286416b452a8de92d69237c5b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67e69286416b452a8de92d69237c5b08, State: Initialized, Role: FOLLOWER
I20260924 11:33:56.357949 9348 consensus_queue.cc:260] T cde220d6c53e4bb99a53fd3f00f05e2c P 67e69286416b452a8de92d69237c5b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } }
I20260924 11:33:56.358521 9348 ts_tablet_manager.cc:1428] T cde220d6c53e4bb99a53fd3f00f05e2c P 67e69286416b452a8de92d69237c5b08: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260924 11:33:56.447664 9329 raft_consensus.cc:482] T cde220d6c53e4bb99a53fd3f00f05e2c P 5c9aac6a3bab408fb9764d14a9b48903 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:33:56.447858 9329 raft_consensus.cc:504] T cde220d6c53e4bb99a53fd3f00f05e2c P 5c9aac6a3bab408fb9764d14a9b48903 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } }
I20260924 11:33:56.448522 9329 leader_election.cc:288] T cde220d6c53e4bb99a53fd3f00f05e2c P 5c9aac6a3bab408fb9764d14a9b48903 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67e69286416b452a8de92d69237c5b08 (127.6.212.2:45989), 1d9ae86a5c17417c8980ccfc29b037d0 (127.6.212.3:38443)
I20260924 11:33:56.449106 9182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cde220d6c53e4bb99a53fd3f00f05e2c" candidate_uuid: "5c9aac6a3bab408fb9764d14a9b48903" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" is_pre_election: true
I20260924 11:33:56.449105 9046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cde220d6c53e4bb99a53fd3f00f05e2c" candidate_uuid: "5c9aac6a3bab408fb9764d14a9b48903" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e69286416b452a8de92d69237c5b08" is_pre_election: true
I20260924 11:33:56.449364 9046 raft_consensus.cc:2454] T cde220d6c53e4bb99a53fd3f00f05e2c P 67e69286416b452a8de92d69237c5b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c9aac6a3bab408fb9764d14a9b48903 in term 0.
I20260924 11:33:56.449385 9182 raft_consensus.cc:2454] T cde220d6c53e4bb99a53fd3f00f05e2c P 1d9ae86a5c17417c8980ccfc29b037d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c9aac6a3bab408fb9764d14a9b48903 in term 0.
I20260924 11:33:56.449833 8836 leader_election.cc:302] T cde220d6c53e4bb99a53fd3f00f05e2c P 5c9aac6a3bab408fb9764d14a9b48903 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1d9ae86a5c17417c8980ccfc29b037d0, 5c9aac6a3bab408fb9764d14a9b48903, 67e69286416b452a8de92d69237c5b08; no voters:
I20260924 11:33:56.450182 9329 raft_consensus.cc:2781] T cde220d6c53e4bb99a53fd3f00f05e2c P 5c9aac6a3bab408fb9764d14a9b48903 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:33:56.450282 9329 raft_consensus.cc:482] T cde220d6c53e4bb99a53fd3f00f05e2c P 5c9aac6a3bab408fb9764d14a9b48903 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:33:56.450397 9329 raft_consensus.cc:3037] T cde220d6c53e4bb99a53fd3f00f05e2c P 5c9aac6a3bab408fb9764d14a9b48903 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:33:56.452483 9329 raft_consensus.cc:504] T cde220d6c53e4bb99a53fd3f00f05e2c P 5c9aac6a3bab408fb9764d14a9b48903 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } }
I20260924 11:33:56.453011 9329 leader_election.cc:288] T cde220d6c53e4bb99a53fd3f00f05e2c P 5c9aac6a3bab408fb9764d14a9b48903 [CANDIDATE]: Term 1 election: Requested vote from peers 67e69286416b452a8de92d69237c5b08 (127.6.212.2:45989), 1d9ae86a5c17417c8980ccfc29b037d0 (127.6.212.3:38443)
I20260924 11:33:56.453475 9046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cde220d6c53e4bb99a53fd3f00f05e2c" candidate_uuid: "5c9aac6a3bab408fb9764d14a9b48903" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e69286416b452a8de92d69237c5b08"
I20260924 11:33:56.453455 9182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cde220d6c53e4bb99a53fd3f00f05e2c" candidate_uuid: "5c9aac6a3bab408fb9764d14a9b48903" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d9ae86a5c17417c8980ccfc29b037d0"
I20260924 11:33:56.453670 9046 raft_consensus.cc:3037] T cde220d6c53e4bb99a53fd3f00f05e2c P 67e69286416b452a8de92d69237c5b08 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:33:56.453680 9182 raft_consensus.cc:3037] T cde220d6c53e4bb99a53fd3f00f05e2c P 1d9ae86a5c17417c8980ccfc29b037d0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:33:56.456040 9046 raft_consensus.cc:2454] T cde220d6c53e4bb99a53fd3f00f05e2c P 67e69286416b452a8de92d69237c5b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c9aac6a3bab408fb9764d14a9b48903 in term 1.
I20260924 11:33:56.456058 9182 raft_consensus.cc:2454] T cde220d6c53e4bb99a53fd3f00f05e2c P 1d9ae86a5c17417c8980ccfc29b037d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c9aac6a3bab408fb9764d14a9b48903 in term 1.
I20260924 11:33:56.456485 8836 leader_election.cc:302] T cde220d6c53e4bb99a53fd3f00f05e2c P 5c9aac6a3bab408fb9764d14a9b48903 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c9aac6a3bab408fb9764d14a9b48903, 67e69286416b452a8de92d69237c5b08; no voters:
I20260924 11:33:56.456815 9329 raft_consensus.cc:2781] T cde220d6c53e4bb99a53fd3f00f05e2c P 5c9aac6a3bab408fb9764d14a9b48903 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:33:56.456980 9329 raft_consensus.cc:686] T cde220d6c53e4bb99a53fd3f00f05e2c P 5c9aac6a3bab408fb9764d14a9b48903 [term 1 LEADER]: Becoming Leader. State: Replica: 5c9aac6a3bab408fb9764d14a9b48903, State: Running, Role: LEADER
I20260924 11:33:56.457257 9329 consensus_queue.cc:237] T cde220d6c53e4bb99a53fd3f00f05e2c P 5c9aac6a3bab408fb9764d14a9b48903 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } }
I20260924 11:33:56.459913 8748 catalog_manager.cc:5495] T cde220d6c53e4bb99a53fd3f00f05e2c P 5c9aac6a3bab408fb9764d14a9b48903 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c9aac6a3bab408fb9764d14a9b48903 (127.6.212.1). New cstate: current_term: 1 leader_uuid: "5c9aac6a3bab408fb9764d14a9b48903" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } health_report { overall_health: UNKNOWN } } }
I20260924 11:33:56.482156 9355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:56.476612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36468 (local address 127.6.212.1:33699)
0924 11:33:56.477132 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:56.477136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:56.477156 (+ 20us) server_negotiation.cc:407] Connection header received
0924 11:33:56.477204 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:56.477207 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:56.477262 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:56.477342 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:33:56.477692 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:56.478838 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:56.480385 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:56.480790 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:56.480880 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:56.481334 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:33:56.481435 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:33:56.481503 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:56.481690 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 11:33:56.481806 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":302,"threads_started":1}
I20260924 11:33:56.491456 9182 raft_consensus.cc:1261] T cde220d6c53e4bb99a53fd3f00f05e2c P 1d9ae86a5c17417c8980ccfc29b037d0 [term 1 FOLLOWER]: Refusing update from remote peer 5c9aac6a3bab408fb9764d14a9b48903: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:33:56.491604 9046 raft_consensus.cc:1261] T cde220d6c53e4bb99a53fd3f00f05e2c P 67e69286416b452a8de92d69237c5b08 [term 1 FOLLOWER]: Refusing update from remote peer 5c9aac6a3bab408fb9764d14a9b48903: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:33:56.492079 9329 consensus_queue.cc:1035] T cde220d6c53e4bb99a53fd3f00f05e2c P 5c9aac6a3bab408fb9764d14a9b48903 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:33:56.492347 9303 consensus_queue.cc:1035] T cde220d6c53e4bb99a53fd3f00f05e2c P 5c9aac6a3bab408fb9764d14a9b48903 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:33:56.531947 8747 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:33:56.532152 8747 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:33:56.534814 8747 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fa7becf8bf9b4da3b0b57bc5dcadbc18: Sending DeleteTablet for 3 replicas of tablet cde220d6c53e4bb99a53fd3f00f05e2c
I20260924 11:33:56.535470 8888 tablet_service.cc:1514] Processing DeleteTablet for tablet cde220d6c53e4bb99a53fd3f00f05e2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:33:56 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:53276
I20260924 11:33:56.535542 9023 tablet_service.cc:1514] Processing DeleteTablet for tablet cde220d6c53e4bb99a53fd3f00f05e2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:33:56 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:42920
I20260924 11:33:56.535748 9335 tablet_replica.cc:330] T cde220d6c53e4bb99a53fd3f00f05e2c P 5c9aac6a3bab408fb9764d14a9b48903: stopping tablet replica
I20260924 11:33:56.535687 9163 tablet_service.cc:1514] Processing DeleteTablet for tablet cde220d6c53e4bb99a53fd3f00f05e2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:33:56 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:51384
I20260924 11:33:56.535983 9335 raft_consensus.cc:2229] T cde220d6c53e4bb99a53fd3f00f05e2c P 5c9aac6a3bab408fb9764d14a9b48903 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:33:56.536028 9337 tablet_replica.cc:330] T cde220d6c53e4bb99a53fd3f00f05e2c P 1d9ae86a5c17417c8980ccfc29b037d0: stopping tablet replica
I20260924 11:33:56.536079 9336 tablet_replica.cc:330] T cde220d6c53e4bb99a53fd3f00f05e2c P 67e69286416b452a8de92d69237c5b08: stopping tablet replica
I20260924 11:33:56.536197 9337 raft_consensus.cc:2229] T cde220d6c53e4bb99a53fd3f00f05e2c P 1d9ae86a5c17417c8980ccfc29b037d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:33:56.536298 9336 raft_consensus.cc:2229] T cde220d6c53e4bb99a53fd3f00f05e2c P 67e69286416b452a8de92d69237c5b08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:33:56.536310 9335 raft_consensus.cc:2258] T cde220d6c53e4bb99a53fd3f00f05e2c P 5c9aac6a3bab408fb9764d14a9b48903 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:33:56.536381 9337 raft_consensus.cc:2258] T cde220d6c53e4bb99a53fd3f00f05e2c P 1d9ae86a5c17417c8980ccfc29b037d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:33:56.536485 9336 raft_consensus.cc:2258] T cde220d6c53e4bb99a53fd3f00f05e2c P 67e69286416b452a8de92d69237c5b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:33:56.536792 8747 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48036:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:33:56.537441 8747 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:33:56.537545 9337 ts_tablet_manager.cc:1905] T cde220d6c53e4bb99a53fd3f00f05e2c P 1d9ae86a5c17417c8980ccfc29b037d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:33:56.537556 9336 ts_tablet_manager.cc:1905] T cde220d6c53e4bb99a53fd3f00f05e2c P 67e69286416b452a8de92d69237c5b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:33:56.537556 9335 ts_tablet_manager.cc:1905] T cde220d6c53e4bb99a53fd3f00f05e2c P 5c9aac6a3bab408fb9764d14a9b48903: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:33:56.544014 9335 ts_tablet_manager.cc:1918] T cde220d6c53e4bb99a53fd3f00f05e2c P 5c9aac6a3bab408fb9764d14a9b48903: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 11:33:56.544014 9336 ts_tablet_manager.cc:1918] T cde220d6c53e4bb99a53fd3f00f05e2c P 67e69286416b452a8de92d69237c5b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 11:33:56.544171 9335 log.cc:1196] T cde220d6c53e4bb99a53fd3f00f05e2c P 5c9aac6a3bab408fb9764d14a9b48903: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-0/wal/wals/cde220d6c53e4bb99a53fd3f00f05e2c
I20260924 11:33:56.544181 9336 log.cc:1196] T cde220d6c53e4bb99a53fd3f00f05e2c P 67e69286416b452a8de92d69237c5b08: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-1/wal/wals/cde220d6c53e4bb99a53fd3f00f05e2c
I20260924 11:33:56.544588 9335 ts_tablet_manager.cc:1939] T cde220d6c53e4bb99a53fd3f00f05e2c P 5c9aac6a3bab408fb9764d14a9b48903: Deleting consensus metadata
I20260924 11:33:56.544588 9336 ts_tablet_manager.cc:1939] T cde220d6c53e4bb99a53fd3f00f05e2c P 67e69286416b452a8de92d69237c5b08: Deleting consensus metadata
I20260924 11:33:56.544657 9163 tablet_service.cc:1467] Processing CreateTablet for tablet d8383f10b1eb4b1c8d2dd8156dc415a9 (DEFAULT_TABLE table=test-table [id=9b31312d5e014cd491f1303271740819]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:33:56.545089 9163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8383f10b1eb4b1c8d2dd8156dc415a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:33:56.545552 8733 catalog_manager.cc:4841] TS 67e69286416b452a8de92d69237c5b08 (127.6.212.2:45989): tablet cde220d6c53e4bb99a53fd3f00f05e2c (table test-table [id=9ff98c7ee0904aa895111e0d21cbe2c6]) successfully deleted
I20260924 11:33:56.545715 8888 tablet_service.cc:1467] Processing CreateTablet for tablet d8383f10b1eb4b1c8d2dd8156dc415a9 (DEFAULT_TABLE table=test-table [id=9b31312d5e014cd491f1303271740819]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:33:56.546072 8888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8383f10b1eb4b1c8d2dd8156dc415a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:33:56.551565 9347 tablet_bootstrap.cc:492] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 1d9ae86a5c17417c8980ccfc29b037d0: Bootstrap starting.
I20260924 11:33:56.551595 8735 catalog_manager.cc:4841] TS 5c9aac6a3bab408fb9764d14a9b48903 (127.6.212.1:33699): tablet cde220d6c53e4bb99a53fd3f00f05e2c (table test-table [id=9ff98c7ee0904aa895111e0d21cbe2c6]) successfully deleted
I20260924 11:33:56.551811 9337 ts_tablet_manager.cc:1918] T cde220d6c53e4bb99a53fd3f00f05e2c P 1d9ae86a5c17417c8980ccfc29b037d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 11:33:56.551954 9337 log.cc:1196] T cde220d6c53e4bb99a53fd3f00f05e2c P 1d9ae86a5c17417c8980ccfc29b037d0: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-2/wal/wals/cde220d6c53e4bb99a53fd3f00f05e2c
I20260924 11:33:56.551903 9023 tablet_service.cc:1467] Processing CreateTablet for tablet d8383f10b1eb4b1c8d2dd8156dc415a9 (DEFAULT_TABLE table=test-table [id=9b31312d5e014cd491f1303271740819]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:33:56.552302 9023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8383f10b1eb4b1c8d2dd8156dc415a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:33:56.552429 9337 ts_tablet_manager.cc:1939] T cde220d6c53e4bb99a53fd3f00f05e2c P 1d9ae86a5c17417c8980ccfc29b037d0: Deleting consensus metadata
I20260924 11:33:56.552760 9346 tablet_bootstrap.cc:492] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 5c9aac6a3bab408fb9764d14a9b48903: Bootstrap starting.
I20260924 11:33:56.553397 9347 tablet_bootstrap.cc:654] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 1d9ae86a5c17417c8980ccfc29b037d0: Neither blocks nor log segments found. Creating new log.
I20260924 11:33:56.553910 8734 catalog_manager.cc:4841] TS 1d9ae86a5c17417c8980ccfc29b037d0 (127.6.212.3:38443): tablet cde220d6c53e4bb99a53fd3f00f05e2c (table test-table [id=9ff98c7ee0904aa895111e0d21cbe2c6]) successfully deleted
I20260924 11:33:56.554322 9346 tablet_bootstrap.cc:654] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 5c9aac6a3bab408fb9764d14a9b48903: Neither blocks nor log segments found. Creating new log.
I20260924 11:33:56.556013 9347 tablet_bootstrap.cc:492] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 1d9ae86a5c17417c8980ccfc29b037d0: No bootstrap required, opened a new log
I20260924 11:33:56.556308 9347 ts_tablet_manager.cc:1397] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 1d9ae86a5c17417c8980ccfc29b037d0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 11:33:56.556677 9346 tablet_bootstrap.cc:492] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 5c9aac6a3bab408fb9764d14a9b48903: No bootstrap required, opened a new log
I20260924 11:33:56.556825 9346 ts_tablet_manager.cc:1397] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 5c9aac6a3bab408fb9764d14a9b48903: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260924 11:33:56.557057 9347 raft_consensus.cc:348] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 1d9ae86a5c17417c8980ccfc29b037d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } }
I20260924 11:33:56.557271 9347 raft_consensus.cc:374] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 1d9ae86a5c17417c8980ccfc29b037d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:33:56.557353 9347 raft_consensus.cc:729] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 1d9ae86a5c17417c8980ccfc29b037d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d9ae86a5c17417c8980ccfc29b037d0, State: Initialized, Role: FOLLOWER
I20260924 11:33:56.557534 9346 raft_consensus.cc:348] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 5c9aac6a3bab408fb9764d14a9b48903 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } }
I20260924 11:33:56.557597 9347 consensus_queue.cc:260] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 1d9ae86a5c17417c8980ccfc29b037d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } }
I20260924 11:33:56.558079 9347 ts_tablet_manager.cc:1428] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 1d9ae86a5c17417c8980ccfc29b037d0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260924 11:33:56.557781 9346 raft_consensus.cc:374] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 5c9aac6a3bab408fb9764d14a9b48903 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:33:56.558490 9346 raft_consensus.cc:729] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 5c9aac6a3bab408fb9764d14a9b48903 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c9aac6a3bab408fb9764d14a9b48903, State: Initialized, Role: FOLLOWER
I20260924 11:33:56.558733 9346 consensus_queue.cc:260] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 5c9aac6a3bab408fb9764d14a9b48903 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } }
I20260924 11:33:56.559799 9346 ts_tablet_manager.cc:1428] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 5c9aac6a3bab408fb9764d14a9b48903: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 11:33:56.560264 9348 tablet_bootstrap.cc:492] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 67e69286416b452a8de92d69237c5b08: Bootstrap starting.
I20260924 11:33:56.561926 9348 tablet_bootstrap.cc:654] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 67e69286416b452a8de92d69237c5b08: Neither blocks nor log segments found. Creating new log.
I20260924 11:33:56.564088 9348 tablet_bootstrap.cc:492] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 67e69286416b452a8de92d69237c5b08: No bootstrap required, opened a new log
I20260924 11:33:56.564260 9348 ts_tablet_manager.cc:1397] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 67e69286416b452a8de92d69237c5b08: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260924 11:33:56.564927 9348 raft_consensus.cc:348] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 67e69286416b452a8de92d69237c5b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } }
I20260924 11:33:56.565143 9348 raft_consensus.cc:374] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 67e69286416b452a8de92d69237c5b08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:33:56.565246 9348 raft_consensus.cc:729] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 67e69286416b452a8de92d69237c5b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67e69286416b452a8de92d69237c5b08, State: Initialized, Role: FOLLOWER
I20260924 11:33:56.565491 9348 consensus_queue.cc:260] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 67e69286416b452a8de92d69237c5b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } }
I20260924 11:33:56.565945 9348 ts_tablet_manager.cc:1428] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 67e69286416b452a8de92d69237c5b08: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260924 11:33:56.804116 9352 raft_consensus.cc:482] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 1d9ae86a5c17417c8980ccfc29b037d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:33:56.804322 9352 raft_consensus.cc:504] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 1d9ae86a5c17417c8980ccfc29b037d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } }
I20260924 11:33:56.805953 9352 leader_election.cc:288] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 1d9ae86a5c17417c8980ccfc29b037d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c9aac6a3bab408fb9764d14a9b48903 (127.6.212.1:33699), 67e69286416b452a8de92d69237c5b08 (127.6.212.2:45989)
I20260924 11:33:56.814772 9355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:56.806324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:46357 (local address 127.6.212.1:33699)
0924 11:33:56.806556 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:56.806565 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:56.807399 (+ 834us) server_negotiation.cc:407] Connection header received
0924 11:33:56.807631 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:56.807635 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:56.807722 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:56.807831 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:33:56.808478 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:56.809612 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:56.812325 (+ 2713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:56.813643 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:56.813771 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:56.813875 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:56.814389 (+ 514us) server_negotiation.cc:299] Negotiation successful
0924 11:33:56.814519 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 11:33:56.814875 9365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:56.806339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:51383 (local address 127.6.212.2:45989)
0924 11:33:56.807110 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:56.807119 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:56.807710 (+ 591us) server_negotiation.cc:407] Connection header received
0924 11:33:56.807857 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:56.807863 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:56.807945 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:56.808043 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:33:56.808661 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:56.809821 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:56.812325 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:56.813643 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:56.813771 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:56.813884 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:56.814389 (+ 505us) server_negotiation.cc:299] Negotiation successful
0924 11:33:56.814520 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":386,"threads_started":1}
I20260924 11:33:56.815426 9363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:56.806137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:33699 (local address 127.6.212.3:46357)
0924 11:33:56.807267 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0924 11:33:56.807286 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:33:56.807480 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:33:56.807898 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:33:56.807902 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:33:56.807927 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:33:56.808261 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:56.808272 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:56.809759 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:56.809767 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:56.812146 (+ 2379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:56.812157 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:56.813921 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:56.813926 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:56.814184 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:56.814239 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 11:33:56.814332 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 11:33:56.814492 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":996,"mutex_wait_us":131,"thread_start_us":1213,"threads_started":1}
I20260924 11:33:56.815791 9364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:56.806180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.2:45989 (local address 127.6.212.3:51383)
0924 11:33:56.807616 (+ 1436us) negotiation.cc:107] Waiting for socket to connect
0924 11:33:56.807630 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 11:33:56.807733 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:33:56.808059 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:33:56.808063 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:33:56.808086 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:33:56.808516 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:56.808524 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:56.809934 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:56.809939 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:56.812143 (+ 2204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:56.812157 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:56.813916 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:56.813923 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:56.814163 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:56.814236 (+ 73us) client_negotiation.cc:769] Sending connection context
0924 11:33:56.814332 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 11:33:56.814492 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1340,"mutex_wait_us":67,"thread_start_us":812,"threads_started":1}
I20260924 11:33:56.816166 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8383f10b1eb4b1c8d2dd8156dc415a9" candidate_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c9aac6a3bab408fb9764d14a9b48903" is_pre_election: true
I20260924 11:33:56.816432 8910 raft_consensus.cc:2454] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 5c9aac6a3bab408fb9764d14a9b48903 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d9ae86a5c17417c8980ccfc29b037d0 in term 0.
I20260924 11:33:56.816896 9117 leader_election.cc:302] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 1d9ae86a5c17417c8980ccfc29b037d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d9ae86a5c17417c8980ccfc29b037d0, 5c9aac6a3bab408fb9764d14a9b48903; no voters:
I20260924 11:33:56.817144 9046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8383f10b1eb4b1c8d2dd8156dc415a9" candidate_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e69286416b452a8de92d69237c5b08" is_pre_election: true
I20260924 11:33:56.817378 9352 raft_consensus.cc:2781] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 1d9ae86a5c17417c8980ccfc29b037d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:33:56.817415 9046 raft_consensus.cc:2454] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 67e69286416b452a8de92d69237c5b08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d9ae86a5c17417c8980ccfc29b037d0 in term 0.
I20260924 11:33:56.817520 9352 raft_consensus.cc:482] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 1d9ae86a5c17417c8980ccfc29b037d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:33:56.817638 9352 raft_consensus.cc:3037] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 1d9ae86a5c17417c8980ccfc29b037d0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:33:56.819871 9352 raft_consensus.cc:504] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 1d9ae86a5c17417c8980ccfc29b037d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } }
I20260924 11:33:56.820411 9352 leader_election.cc:288] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 1d9ae86a5c17417c8980ccfc29b037d0 [CANDIDATE]: Term 1 election: Requested vote from peers 5c9aac6a3bab408fb9764d14a9b48903 (127.6.212.1:33699), 67e69286416b452a8de92d69237c5b08 (127.6.212.2:45989)
I20260924 11:33:56.820852 9046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8383f10b1eb4b1c8d2dd8156dc415a9" candidate_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67e69286416b452a8de92d69237c5b08"
I20260924 11:33:56.821080 9046 raft_consensus.cc:3037] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 67e69286416b452a8de92d69237c5b08 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:33:56.821092 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8383f10b1eb4b1c8d2dd8156dc415a9" candidate_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c9aac6a3bab408fb9764d14a9b48903"
I20260924 11:33:56.821305 8910 raft_consensus.cc:3037] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 5c9aac6a3bab408fb9764d14a9b48903 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:33:56.823072 9046 raft_consensus.cc:2454] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 67e69286416b452a8de92d69237c5b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d9ae86a5c17417c8980ccfc29b037d0 in term 1.
I20260924 11:33:56.823324 8910 raft_consensus.cc:2454] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 5c9aac6a3bab408fb9764d14a9b48903 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d9ae86a5c17417c8980ccfc29b037d0 in term 1.
I20260924 11:33:56.823520 9115 leader_election.cc:302] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 1d9ae86a5c17417c8980ccfc29b037d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d9ae86a5c17417c8980ccfc29b037d0, 67e69286416b452a8de92d69237c5b08; no voters:
I20260924 11:33:56.823823 9352 raft_consensus.cc:2781] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 1d9ae86a5c17417c8980ccfc29b037d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:33:56.824255 9352 raft_consensus.cc:686] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 1d9ae86a5c17417c8980ccfc29b037d0 [term 1 LEADER]: Becoming Leader. State: Replica: 1d9ae86a5c17417c8980ccfc29b037d0, State: Running, Role: LEADER
I20260924 11:33:56.824617 9352 consensus_queue.cc:237] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 1d9ae86a5c17417c8980ccfc29b037d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } } peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } }
I20260924 11:33:56.827967 8747 catalog_manager.cc:5495] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 1d9ae86a5c17417c8980ccfc29b037d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d9ae86a5c17417c8980ccfc29b037d0 (127.6.212.3). New cstate: current_term: 1 leader_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d9ae86a5c17417c8980ccfc29b037d0" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 38443 } health_report { overall_health: HEALTHY } } }
I20260924 11:33:56.869347 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:56.863134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56460 (local address 127.6.212.3:38443)
0924 11:33:56.864177 (+ 1043us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:56.864184 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:56.864206 (+ 22us) server_negotiation.cc:407] Connection header received
0924 11:33:56.864323 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:56.864327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:56.864398 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:56.864572 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:33:56.865006 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:56.865755 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:56.867083 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:56.867554 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:56.867685 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:56.868121 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:33:56.868688 (+ 567us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:33:56.868864 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:56.868940 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 11:33:56.869016 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":908,"thread_start_us":789,"threads_started":1}
I20260924 11:33:56.884418 9365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:56.879405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:43695 (local address 127.6.212.2:45989)
0924 11:33:56.879666 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:56.879674 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:56.879803 (+ 129us) server_negotiation.cc:407] Connection header received
0924 11:33:56.880030 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:56.880036 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:56.880109 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:56.880196 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:33:56.880658 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:56.881374 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:56.882716 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:56.883119 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:56.883259 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:56.883727 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:33:56.883824 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:33:56.883919 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:56.884052 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 11:33:56.884130 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260924 11:33:56.884511 9371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:56.878996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.2:45989 (local address 127.6.212.3:43695)
0924 11:33:56.879699 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0924 11:33:56.879719 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 11:33:56.879901 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:33:56.880220 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:33:56.880224 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:33:56.880235 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 11:33:56.880503 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:56.880511 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:56.881514 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:56.881519 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:56.882575 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:56.882585 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:56.883285 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:56.883290 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:56.883491 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:56.883547 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 11:33:56.883920 (+ 373us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 11:33:56.883933 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 11:33:56.884033 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 11:33:56.884157 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":250,"threads_started":1}
I20260924 11:33:56.892441 9046 raft_consensus.cc:1261] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 67e69286416b452a8de92d69237c5b08 [term 1 FOLLOWER]: Refusing update from remote peer 1d9ae86a5c17417c8980ccfc29b037d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:33:56.892432 8910 raft_consensus.cc:1261] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 5c9aac6a3bab408fb9764d14a9b48903 [term 1 FOLLOWER]: Refusing update from remote peer 1d9ae86a5c17417c8980ccfc29b037d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:33:56.893162 9366 consensus_queue.cc:1035] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 1d9ae86a5c17417c8980ccfc29b037d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67e69286416b452a8de92d69237c5b08" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:33:56.893541 9352 consensus_queue.cc:1035] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 1d9ae86a5c17417c8980ccfc29b037d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c9aac6a3bab408fb9764d14a9b48903" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 33699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:33:56.918972 9376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:56.913168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.3:38443 (local address 127.6.212.2:45303)
0924 11:33:56.914212 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0924 11:33:56.914228 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:33:56.914336 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:33:56.914590 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:33:56.914593 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:33:56.914602 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 11:33:56.914869 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:56.914877 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:56.915954 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:56.915959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:56.917427 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:33:56.917440 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:33:56.918080 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:33:56.918084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:33:56.918190 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:56.918223 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 11:33:56.918529 (+ 306us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 11:33:56.918537 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 11:33:56.918592 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 11:33:56.918681 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":895,"thread_start_us":771,"threads_started":1}
I20260924 11:33:56.918962 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:56.913789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:45303 (local address 127.6.212.3:38443)
0924 11:33:56.913981 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:56.913987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:56.914305 (+ 318us) server_negotiation.cc:407] Connection header received
0924 11:33:56.914431 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:56.914435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:56.914492 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:56.914563 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:33:56.914987 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:56.915827 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:56.917562 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:56.917940 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:56.918033 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:56.918350 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:33:56.918450 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:33:56.918514 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:56.918628 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 11:33:56.918682 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260924 11:33:56.920185 9317 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 11:33:58.523416 8748 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:33:58.523661 8748 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:33:58.527489 8748 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fa7becf8bf9b4da3b0b57bc5dcadbc18: Sending DeleteTablet for 3 replicas of tablet d8383f10b1eb4b1c8d2dd8156dc415a9
I20260924 11:33:58.528273 8888 tablet_service.cc:1514] Processing DeleteTablet for tablet d8383f10b1eb4b1c8d2dd8156dc415a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:33:58 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:53276
I20260924 11:33:58.528273 9023 tablet_service.cc:1514] Processing DeleteTablet for tablet d8383f10b1eb4b1c8d2dd8156dc415a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:33:58 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:42920
I20260924 11:33:58.528309 9163 tablet_service.cc:1514] Processing DeleteTablet for tablet d8383f10b1eb4b1c8d2dd8156dc415a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:33:58 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:51384
I20260924 11:33:58.528962 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 8819
I20260924 11:33:58.540840 9400 tablet_replica.cc:330] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 5c9aac6a3bab408fb9764d14a9b48903: stopping tablet replica
I20260924 11:33:58.541296 9400 raft_consensus.cc:2229] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 5c9aac6a3bab408fb9764d14a9b48903 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:33:58.541740 9400 raft_consensus.cc:2258] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 5c9aac6a3bab408fb9764d14a9b48903 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:33:58.542529 9401 tablet_replica.cc:330] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 67e69286416b452a8de92d69237c5b08: stopping tablet replica
I20260924 11:33:58.542824 9402 tablet_replica.cc:330] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 1d9ae86a5c17417c8980ccfc29b037d0: stopping tablet replica
I20260924 11:33:58.543170 9402 raft_consensus.cc:2229] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 1d9ae86a5c17417c8980ccfc29b037d0 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:33:58.543320 9401 raft_consensus.cc:2229] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 67e69286416b452a8de92d69237c5b08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:33:58.543388 9400 ts_tablet_manager.cc:1905] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 5c9aac6a3bab408fb9764d14a9b48903: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:33:58.543646 9401 raft_consensus.cc:2258] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 67e69286416b452a8de92d69237c5b08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:33:58.543803 9402 raft_consensus.cc:2258] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 1d9ae86a5c17417c8980ccfc29b037d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:33:58.544720 9401 ts_tablet_manager.cc:1905] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 67e69286416b452a8de92d69237c5b08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:33:58.545127 9402 ts_tablet_manager.cc:1905] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 1d9ae86a5c17417c8980ccfc29b037d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:33:58.550729 9400 ts_tablet_manager.cc:1918] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 5c9aac6a3bab408fb9764d14a9b48903: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 11:33:58.550889 9400 log.cc:1196] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 5c9aac6a3bab408fb9764d14a9b48903: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-0/wal/wals/d8383f10b1eb4b1c8d2dd8156dc415a9
I20260924 11:33:58.551283 9400 ts_tablet_manager.cc:1939] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 5c9aac6a3bab408fb9764d14a9b48903: Deleting consensus metadata
I20260924 11:33:58.551599 9401 ts_tablet_manager.cc:1918] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 67e69286416b452a8de92d69237c5b08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 11:33:58.551750 9401 log.cc:1196] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 67e69286416b452a8de92d69237c5b08: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-1/wal/wals/d8383f10b1eb4b1c8d2dd8156dc415a9
I20260924 11:33:58.552220 9401 ts_tablet_manager.cc:1939] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 67e69286416b452a8de92d69237c5b08: Deleting consensus metadata
I20260924 11:33:58.552441 9402 ts_tablet_manager.cc:1918] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 1d9ae86a5c17417c8980ccfc29b037d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 11:33:58.552592 9402 log.cc:1196] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 1d9ae86a5c17417c8980ccfc29b037d0: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790249617057452-6992-0/minicluster-data/ts-2/wal/wals/d8383f10b1eb4b1c8d2dd8156dc415a9
I20260924 11:33:58.552517 8735 catalog_manager.cc:4841] TS 5c9aac6a3bab408fb9764d14a9b48903 (127.6.212.1:33699): tablet d8383f10b1eb4b1c8d2dd8156dc415a9 (table test-table [id=9b31312d5e014cd491f1303271740819]) successfully deleted
I20260924 11:33:58.553040 9402 ts_tablet_manager.cc:1939] T d8383f10b1eb4b1c8d2dd8156dc415a9 P 1d9ae86a5c17417c8980ccfc29b037d0: Deleting consensus metadata
I20260924 11:33:58.553399 8733 catalog_manager.cc:4841] TS 67e69286416b452a8de92d69237c5b08 (127.6.212.2:45989): tablet d8383f10b1eb4b1c8d2dd8156dc415a9 (table test-table [id=9b31312d5e014cd491f1303271740819]) successfully deleted
I20260924 11:33:58.554013 8734 catalog_manager.cc:4841] TS 1d9ae86a5c17417c8980ccfc29b037d0 (127.6.212.3:38443): tablet d8383f10b1eb4b1c8d2dd8156dc415a9 (table test-table [id=9b31312d5e014cd491f1303271740819]) successfully deleted
Sep 24 11:33:58 dist-test-slave-cq7c krb5kdc[8703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249636, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.212.1@KRBTEST.COM
I20260924 11:33:58.567446 9404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:58.531106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36472 (local address 127.6.212.1:33699)
0924 11:33:58.543289 (+ 12183us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:58.543295 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:58.543318 (+ 23us) server_negotiation.cc:407] Connection header received
0924 11:33:58.543405 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:58.543413 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:58.543512 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:58.543620 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:33:58.545639 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:58.547238 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:58.549261 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:58.549933 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:58.550049 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:58.561415 (+ 11366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:33:58.561431 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:33:58.561448 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:33:58.561486 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:33:58.564644 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:58.565344 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:58.565353 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:58.565356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:58.565496 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:58.565846 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:58.565856 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:58.565858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:58.566214 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:33:58.566447 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:58.566813 (+ 366us) server_negotiation.cc:299] Negotiation successful
0924 11:33:58.567053 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12040,"thread_start_us":11922,"threads_started":1}
W20260924 11:33:58.568089 8950 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36472
W20260924 11:33:58.569069 6992 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 8819: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:33:58.569252 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 8819
I20260924 11:33:58.593360 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 8959
Sep 24 11:33:58 dist-test-slave-cq7c krb5kdc[8703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249636, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.212.2@KRBTEST.COM
I20260924 11:33:58.608479 9405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:58.594087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59590 (local address 127.6.212.2:45989)
0924 11:33:58.594655 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:58.594659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:58.594678 (+ 19us) server_negotiation.cc:407] Connection header received
0924 11:33:58.594740 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:58.594744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:58.594803 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:58.594898 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:33:58.596331 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:58.597091 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:58.598314 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:58.598705 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:58.598814 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:58.602939 (+ 4125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:33:58.602952 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:33:58.602969 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:33:58.603001 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:33:58.605866 (+ 2865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:58.606478 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:58.606486 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:58.606488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:58.606649 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:58.607042 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:58.607046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:58.607048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:58.607470 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:33:58.607684 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:58.607953 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 11:33:58.608126 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":331,"threads_started":1}
W20260924 11:33:58.609133 9089 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59590
W20260924 11:33:58.609753 6992 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 8959: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:33:58.609958 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 8959
I20260924 11:33:58.635468 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 9098
Sep 24 11:33:58 dist-test-slave-cq7c krb5kdc[8703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249636, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.212.3@KRBTEST.COM
I20260924 11:33:58.650141 9406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:33:58.636158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56474 (local address 127.6.212.3:38443)
0924 11:33:58.636608 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0924 11:33:58.636612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:33:58.636628 (+ 16us) server_negotiation.cc:407] Connection header received
0924 11:33:58.636680 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:33:58.636683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:33:58.636738 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:33:58.636878 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:33:58.638103 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:58.638849 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:58.640170 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:33:58.640562 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:33:58.640654 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:33:58.644774 (+ 4120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:33:58.644788 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:33:58.644805 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:33:58.644836 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:33:58.647797 (+ 2961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:58.648382 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:58.648388 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:58.648391 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:58.648501 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:33:58.648824 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:33:58.648827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:33:58.648829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:33:58.649168 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:33:58.649369 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0924 11:33:58.649608 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 11:33:58.649777 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":231,"threads_started":1}
W20260924 11:33:58.650820 9228 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56474
W20260924 11:33:58.651539 6992 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 9098: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:33:58.651681 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 9098
I20260924 11:33:58.673509 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 8716
I20260924 11:33:58.674690 8788 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:33:58.789762 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 8716
2026-09-24T11:33:58Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8471 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:33:58 dist-test-slave-cq7c krb5kdc[9410](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:33:58 dist-test-slave-cq7c krb5kdc[9410](info): set up 2 sockets
Sep 24 11:33:58 dist-test-slave-cq7c krb5kdc[9410](info): commencing operation
krb5kdc: starting...
W20260924 11:33:59.991614 6992 mini_kdc.cc:121] Time spent starting KDC: real 1.157s user 0.005s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Sep 24 11:34:00 dist-test-slave-cq7c krb5kdc[9410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:34:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:34:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.212.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
I20260924 11:34:00.183548 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:37475
--webserver_interface=127.6.212.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:35865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:37475
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:34:00.506559 9423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:00.506882 9423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:00.507067 9423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:00.514614 9423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:34:00.514747 9423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:00.514844 9423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:00.514961 9423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:34:00.515027 9423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:34:00.525202 9423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:35865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:37475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:37475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.212.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:34:00.526590 9423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:00.528095 9423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:00.540937 9429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:00.540953 9428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:00.541529 9431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:00.541851 9423 server_base.cc:1034] running on GCE node
I20260924 11:34:00.542766 9423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:00.544360 9423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:00.545588 9423 hybrid_clock.cc:648] HybridClock initialized: now 1790249640545514 us; error 61 us; skew 500 ppm
I20260924 11:34:00.546099 9423 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:34:00 dist-test-slave-cq7c krb5kdc[9410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249640, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:00.549553 9423 init.cc:377] Logged in from keytab as kudu/127.6.212.62@KRBTEST.COM (short username kudu)
I20260924 11:34:00.551191 9423 webserver.cc:460] Webserver started at http://127.6.212.62:42663/ using document root <none> and password file <none>
I20260924 11:34:00.551967 9423 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:00.552083 9423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:00.552366 9423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:34:00.555629 9423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "f4c07532c72a42a4b0146e21c6b8f3fa"
format_stamp: "Formatted at 2026-09-24 11:34:00 on dist-test-slave-cq7c"
server_key: "02fa7f45c9a5338fc94befa0fb75c1dd"
server_key_iv: "88d978930864e1e71a5eeaeea2312c5c"
server_key_version: "encryptionkey@0"
I20260924 11:34:00.556416 9423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "f4c07532c72a42a4b0146e21c6b8f3fa"
format_stamp: "Formatted at 2026-09-24 11:34:00 on dist-test-slave-cq7c"
server_key: "02fa7f45c9a5338fc94befa0fb75c1dd"
server_key_iv: "88d978930864e1e71a5eeaeea2312c5c"
server_key_version: "encryptionkey@0"
I20260924 11:34:00.562564 9423 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 11:34:00.566386 9438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:00.567981 9423 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:34:00.568176 9423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "f4c07532c72a42a4b0146e21c6b8f3fa"
format_stamp: "Formatted at 2026-09-24 11:34:00 on dist-test-slave-cq7c"
server_key: "02fa7f45c9a5338fc94befa0fb75c1dd"
server_key_iv: "88d978930864e1e71a5eeaeea2312c5c"
server_key_version: "encryptionkey@0"
I20260924 11:34:00.568387 9423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:00.584687 9423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:00.589027 9423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:34:00.589377 9423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:00.609805 9423 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.62:37475
I20260924 11:34:00.609825 9489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.62:37475 every 8 connection(s)
I20260924 11:34:00.611467 9423 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:34:00.612530 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 9423
I20260924 11:34:00.612776 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:34:00.613528 6992 external_mini_cluster.cc:1428] Setting key 28d0556fe38f19a5e361c58ad15febf7
I20260924 11:34:00.616380 9490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 11:34:00 dist-test-slave-cq7c krb5kdc[9410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:34:00.629659 9490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4c07532c72a42a4b0146e21c6b8f3fa: Bootstrap starting.
I20260924 11:34:00.632804 9490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4c07532c72a42a4b0146e21c6b8f3fa: Neither blocks nor log segments found. Creating new log.
I20260924 11:34:00.633934 9490 log.cc:824] T 00000000000000000000000000000000 P f4c07532c72a42a4b0146e21c6b8f3fa: Log is configured to *not* fsync() on all Append() calls
I20260924 11:34:00.637027 9490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4c07532c72a42a4b0146e21c6b8f3fa: No bootstrap required, opened a new log
I20260924 11:34:00.638528 9493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:00.615743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42062 (local address 127.6.212.62:37475)
0924 11:34:00.617165 (+ 1422us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:00.617177 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:00.617222 (+ 45us) server_negotiation.cc:407] Connection header received
0924 11:34:00.618929 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:00.618950 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:00.619517 (+ 567us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:00.619883 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:00.621560 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:00.623058 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:00.624820 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:00.625662 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:00.625985 (+ 323us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:00.630556 (+ 4571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:00.630565 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:00.630580 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:00.630624 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:00.634095 (+ 3471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:00.634858 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:00.634897 (+ 39us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:00.634907 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:00.635098 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:00.635602 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:00.635611 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:00.635616 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:00.636130 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:00.636446 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:00.636899 (+ 453us) server_negotiation.cc:299] Negotiation successful
0924 11:34:00.637361 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":423,"threads_started":1}
I20260924 11:34:00.646106 9490 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4c07532c72a42a4b0146e21c6b8f3fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c07532c72a42a4b0146e21c6b8f3fa" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 37475 } }
I20260924 11:34:00.646477 9490 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4c07532c72a42a4b0146e21c6b8f3fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:34:00.646596 9490 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4c07532c72a42a4b0146e21c6b8f3fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4c07532c72a42a4b0146e21c6b8f3fa, State: Initialized, Role: FOLLOWER
I20260924 11:34:00.647207 9490 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4c07532c72a42a4b0146e21c6b8f3fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c07532c72a42a4b0146e21c6b8f3fa" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 37475 } }
I20260924 11:34:00.647495 9490 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4c07532c72a42a4b0146e21c6b8f3fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:34:00.647651 9490 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4c07532c72a42a4b0146e21c6b8f3fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:34:00.647825 9490 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4c07532c72a42a4b0146e21c6b8f3fa [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:34:00.742610 9490 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4c07532c72a42a4b0146e21c6b8f3fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c07532c72a42a4b0146e21c6b8f3fa" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 37475 } }
I20260924 11:34:00.743384 9490 leader_election.cc:302] T 00000000000000000000000000000000 P f4c07532c72a42a4b0146e21c6b8f3fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4c07532c72a42a4b0146e21c6b8f3fa; no voters:
I20260924 11:34:00.744267 9495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4c07532c72a42a4b0146e21c6b8f3fa [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:34:00.744454 9490 leader_election.cc:288] T 00000000000000000000000000000000 P f4c07532c72a42a4b0146e21c6b8f3fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:34:00.748324 9495 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4c07532c72a42a4b0146e21c6b8f3fa [term 1 LEADER]: Becoming Leader. State: Replica: f4c07532c72a42a4b0146e21c6b8f3fa, State: Running, Role: LEADER
I20260924 11:34:00.748950 9495 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4c07532c72a42a4b0146e21c6b8f3fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c07532c72a42a4b0146e21c6b8f3fa" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 37475 } }
I20260924 11:34:00.749449 9490 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4c07532c72a42a4b0146e21c6b8f3fa [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:34:00.752120 9496 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4c07532c72a42a4b0146e21c6b8f3fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4c07532c72a42a4b0146e21c6b8f3fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c07532c72a42a4b0146e21c6b8f3fa" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 37475 } } }
I20260924 11:34:00.752081 9497 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4c07532c72a42a4b0146e21c6b8f3fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4c07532c72a42a4b0146e21c6b8f3fa. Latest consensus state: current_term: 1 leader_uuid: "f4c07532c72a42a4b0146e21c6b8f3fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c07532c72a42a4b0146e21c6b8f3fa" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 37475 } } }
I20260924 11:34:00.752490 9496 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4c07532c72a42a4b0146e21c6b8f3fa [sys.catalog]: This master's current role is: LEADER
I20260924 11:34:00.752550 9497 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4c07532c72a42a4b0146e21c6b8f3fa [sys.catalog]: This master's current role is: LEADER
I20260924 11:34:00.753667 9504 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:34:00.763067 9504 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:34:00.773145 9504 catalog_manager.cc:1347] Generated new cluster ID: 2a30a92334ad475283f1d2c8bed67559
I20260924 11:34:00.773281 9504 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:34:00.783147 9504 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:34:00.784502 9504 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:34:00.791162 9504 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4c07532c72a42a4b0146e21c6b8f3fa: Generated new TSK 0
I20260924 11:34:00.792347 9504 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.212.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab.
Entry for principal kudu/127.6.212.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab.
W20260924 11:34:01.934139 6992 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.6.212.1: real 1.137s user 0.000s sys 0.010s
I20260924 11:34:01.935074 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.1:0
--local_ip_for_outbound_sockets=127.6.212.1
--webserver_interface=127.6.212.1
--webserver_port=0
--tserver_master_addrs=127.6.212.62:37475
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab
--principal=kudu/127.6.212.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:35865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:34:02.262368 9516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:02.262674 9516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:02.262770 9516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:02.270068 9516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:02.270195 9516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:02.270313 9516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.1
I20260924 11:34:02.280202 9516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:35865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab
--principal=kudu/127.6.212.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.212.1
--webserver_port=0
--tserver_master_addrs=127.6.212.62:37475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.1
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:34:02.281454 9516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:02.282873 9516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:02.296525 9521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:02.296527 9522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:02.296569 9524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:02.297392 9516 server_base.cc:1034] running on GCE node
I20260924 11:34:02.298031 9516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:02.299120 9516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:02.300344 9516 hybrid_clock.cc:648] HybridClock initialized: now 1790249642300277 us; error 60 us; skew 500 ppm
I20260924 11:34:02.300691 9516 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:34:02 dist-test-slave-cq7c krb5kdc[9410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249642, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:02.304175 9516 init.cc:377] Logged in from keytab as kudu/127.6.212.1@KRBTEST.COM (short username kudu)
I20260924 11:34:02.305660 9516 webserver.cc:460] Webserver started at http://127.6.212.1:41547/ using document root <none> and password file <none>
I20260924 11:34:02.306367 9516 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:02.306519 9516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:02.306840 9516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:34:02.309976 9516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-0/data/instance:
uuid: "cd39e6279cab460ea900cff27cbc17fd"
format_stamp: "Formatted at 2026-09-24 11:34:02 on dist-test-slave-cq7c"
server_key: "5e2533df0e09df77983e90ef791ccc6d"
server_key_iv: "6d91939987598a7f0180b506a68ba5ce"
server_key_version: "encryptionkey@0"
I20260924 11:34:02.310781 9516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance:
uuid: "cd39e6279cab460ea900cff27cbc17fd"
format_stamp: "Formatted at 2026-09-24 11:34:02 on dist-test-slave-cq7c"
server_key: "5e2533df0e09df77983e90ef791ccc6d"
server_key_iv: "6d91939987598a7f0180b506a68ba5ce"
server_key_version: "encryptionkey@0"
I20260924 11:34:02.316464 9516 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 11:34:02.320200 9531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:02.321547 9516 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:34:02.321731 9516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-0/wal
uuid: "cd39e6279cab460ea900cff27cbc17fd"
format_stamp: "Formatted at 2026-09-24 11:34:02 on dist-test-slave-cq7c"
server_key: "5e2533df0e09df77983e90ef791ccc6d"
server_key_iv: "6d91939987598a7f0180b506a68ba5ce"
server_key_version: "encryptionkey@0"
I20260924 11:34:02.321940 9516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:02.347663 9516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:02.352767 9516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:34:02.353075 9516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:02.354206 9516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:34:02.355664 9516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:34:02.355772 9516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:02.355924 9516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:34:02.355996 9516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:02.389674 9516 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.1:42705
I20260924 11:34:02.389726 9643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.1:42705 every 8 connection(s)
I20260924 11:34:02.391072 9516 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
I20260924 11:34:02.394559 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 9516
I20260924 11:34:02.394821 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance
I20260924 11:34:02.395656 6992 external_mini_cluster.cc:1428] Setting key 740f19f52423f55db214bac55336e647
Sep 24 11:34:02 dist-test-slave-cq7c krb5kdc[9410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249642, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.212.2@KRBTEST.COM; defaulting to no policy
I20260924 11:34:02.415297 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:02.393825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:36319 (local address 127.6.212.62:37475)
0924 11:34:02.394440 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:02.394447 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:02.395515 (+ 1068us) server_negotiation.cc:407] Connection header received
0924 11:34:02.398089 (+ 2574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:02.398093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:02.398153 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:02.398240 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:02.400476 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:02.401185 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:02.403830 (+ 2645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:02.404551 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:02.404707 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:02.408707 (+ 4000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:02.408712 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:02.408717 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:02.408751 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:02.411574 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:02.412639 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:02.412644 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:02.412646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:02.412781 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:02.413326 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:02.413330 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:02.413332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:02.413583 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:02.413701 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:02.414721 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0924 11:34:02.414965 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":249,"threads_started":1}
I20260924 11:34:02.416363 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:02.394427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37475 (local address 127.6.212.1:36319)
0924 11:34:02.395307 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:02.395384 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:02.397796 (+ 2412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:02.398537 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:02.398551 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:02.399399 (+ 848us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:02.400282 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:02.400311 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:02.401355 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:02.401363 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:02.403646 (+ 2283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:02.403659 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:02.404783 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:02.404788 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:02.405041 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:02.405764 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:02.405786 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:02.408411 (+ 2625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:02.411726 (+ 3315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:02.411740 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:02.411762 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:02.412494 (+ 732us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:02.412900 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:02.412907 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:02.412913 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:02.413184 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:02.413728 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:02.413737 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:02.414245 (+ 508us) client_negotiation.cc:769] Sending connection context
0924 11:34:02.414611 (+ 366us) client_negotiation.cc:240] Negotiation successful
0924 11:34:02.415218 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":324,"threads_started":1}
I20260924 11:34:02.417876 9644 heartbeater.cc:347] Connected to a master server at 127.6.212.62:37475
I20260924 11:34:02.418283 9644 heartbeater.cc:464] Registering TS with master...
I20260924 11:34:02.419003 9644 heartbeater.cc:511] Master 127.6.212.62:37475 requested a full tablet report, sending...
I20260924 11:34:02.421237 9455 ts_manager.cc:194] Registered new tserver with Master: cd39e6279cab460ea900cff27cbc17fd (127.6.212.1:42705)
I20260924 11:34:02.422708 9455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.1@KRBTEST.COM'} at 127.6.212.1:36319
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.2@KRBTEST.COM" created.
I20260924 11:34:03.425820 9644 heartbeater.cc:503] Master 127.6.212.62:37475 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab.
Entry for principal kudu/127.6.212.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab.
W20260924 11:34:03.513403 6992 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.6.212.2: real 1.115s user 0.001s sys 0.011s
I20260924 11:34:03.514146 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.2:0
--local_ip_for_outbound_sockets=127.6.212.2
--webserver_interface=127.6.212.2
--webserver_port=0
--tserver_master_addrs=127.6.212.62:37475
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab
--principal=kudu/127.6.212.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:35865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:34:03.851897 9650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:03.852187 9650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:03.852341 9650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:03.859882 9650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:03.859993 9650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:03.860184 9650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.2
I20260924 11:34:03.870244 9650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:35865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab
--principal=kudu/127.6.212.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.212.2
--webserver_port=0
--tserver_master_addrs=127.6.212.62:37475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.2
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:34:03.871587 9650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:03.873049 9650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:03.886767 9658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:03.886862 9655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:03.886847 9656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:03.893155 9650 server_base.cc:1034] running on GCE node
I20260924 11:34:03.893842 9650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:03.894788 9650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:03.896047 9650 hybrid_clock.cc:648] HybridClock initialized: now 1790249643895995 us; error 39 us; skew 500 ppm
I20260924 11:34:03.896380 9650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:34:03 dist-test-slave-cq7c krb5kdc[9410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249643, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:03.909300 9650 init.cc:377] Logged in from keytab as kudu/127.6.212.2@KRBTEST.COM (short username kudu)
I20260924 11:34:03.910907 9650 webserver.cc:460] Webserver started at http://127.6.212.2:39897/ using document root <none> and password file <none>
I20260924 11:34:03.911731 9650 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:03.911893 9650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:03.912218 9650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:34:03.915153 9650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-1/data/instance:
uuid: "ed9fdcd4fb054f0f80af6fcce55cbaa7"
format_stamp: "Formatted at 2026-09-24 11:34:03 on dist-test-slave-cq7c"
server_key: "f36d27c3e92af7e78dfc46930bbd3b57"
server_key_iv: "347c2df2dfca51f7a1d08c13297dc317"
server_key_version: "encryptionkey@0"
I20260924 11:34:03.916009 9650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance:
uuid: "ed9fdcd4fb054f0f80af6fcce55cbaa7"
format_stamp: "Formatted at 2026-09-24 11:34:03 on dist-test-slave-cq7c"
server_key: "f36d27c3e92af7e78dfc46930bbd3b57"
server_key_iv: "347c2df2dfca51f7a1d08c13297dc317"
server_key_version: "encryptionkey@0"
I20260924 11:34:03.921838 9650 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 11:34:03.925559 9665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:03.927875 9650 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 11:34:03.928030 9650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-1/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-1/wal
uuid: "ed9fdcd4fb054f0f80af6fcce55cbaa7"
format_stamp: "Formatted at 2026-09-24 11:34:03 on dist-test-slave-cq7c"
server_key: "f36d27c3e92af7e78dfc46930bbd3b57"
server_key_iv: "347c2df2dfca51f7a1d08c13297dc317"
server_key_version: "encryptionkey@0"
I20260924 11:34:03.928216 9650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:03.944341 9650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:03.948777 9650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:34:03.949095 9650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:03.950138 9650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:34:03.951540 9650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:34:03.951643 9650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:03.951797 9650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:34:03.951885 9650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:03.987413 9650 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.2:40795
I20260924 11:34:03.987509 9777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.2:40795 every 8 connection(s)
I20260924 11:34:03.988766 9650 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
I20260924 11:34:03.994217 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 9650
I20260924 11:34:03.994397 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance
I20260924 11:34:03.994912 6992 external_mini_cluster.cc:1428] Setting key d9470de9c300ddcda7d66cb92197117d
Sep 24 11:34:03 dist-test-slave-cq7c krb5kdc[9410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249643, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.2@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:34:04.012712 9780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:03.991537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:44011 (local address 127.6.212.62:37475)
0924 11:34:03.992011 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:03.992015 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:03.992967 (+ 952us) server_negotiation.cc:407] Connection header received
0924 11:34:03.995997 (+ 3030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:03.996003 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:03.996090 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:03.996226 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:03.998198 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:03.998942 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:04.001666 (+ 2724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:04.002108 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:04.002230 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:04.006332 (+ 4102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:04.006338 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:04.006350 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:04.006386 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:04.009114 (+ 2728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:04.010136 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:04.010141 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:04.010144 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:04.010278 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:04.010829 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:04.010834 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:04.010837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:04.011121 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:04.011280 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:04.012119 (+ 839us) server_negotiation.cc:299] Negotiation successful
0924 11:34:04.012357 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":258,"threads_started":1}
WARNING: no policy specified for kudu/127.6.212.3@KRBTEST.COM; defaulting to no policy
I20260924 11:34:04.013832 9781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:03.992118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37475 (local address 127.6.212.2:44011)
0924 11:34:03.992807 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:03.992849 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:03.995710 (+ 2861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:03.996541 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:03.996550 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:03.997182 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:03.998011 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:03.998039 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:03.999105 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:03.999113 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:04.001468 (+ 2355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:04.001487 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:04.002376 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:04.002386 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:04.002741 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:04.003732 (+ 991us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:04.003757 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:04.006061 (+ 2304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:04.009298 (+ 3237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:04.009309 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:04.009330 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:04.009941 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:04.010479 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:04.010484 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:04.010488 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:04.010704 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:04.011259 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:04.011267 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:04.011728 (+ 461us) client_negotiation.cc:769] Sending connection context
0924 11:34:04.012015 (+ 287us) client_negotiation.cc:240] Negotiation successful
0924 11:34:04.012582 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":257,"threads_started":1}
I20260924 11:34:04.015612 9778 heartbeater.cc:347] Connected to a master server at 127.6.212.62:37475
I20260924 11:34:04.015937 9778 heartbeater.cc:464] Registering TS with master...
I20260924 11:34:04.016726 9778 heartbeater.cc:511] Master 127.6.212.62:37475 requested a full tablet report, sending...
I20260924 11:34:04.018209 9455 ts_manager.cc:194] Registered new tserver with Master: ed9fdcd4fb054f0f80af6fcce55cbaa7 (127.6.212.2:40795)
I20260924 11:34:04.019045 9455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.2@KRBTEST.COM'} at 127.6.212.2:44011
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab.
Entry for principal kudu/127.6.212.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab.
W20260924 11:34:04.546905 6992 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.6.212.3: real 0.549s user 0.002s sys 0.008s
I20260924 11:34:04.547827 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.3:0
--local_ip_for_outbound_sockets=127.6.212.3
--webserver_interface=127.6.212.3
--webserver_port=0
--tserver_master_addrs=127.6.212.62:37475
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab
--principal=kudu/127.6.212.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:35865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:34:04.871232 9784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:04.871542 9784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:04.871730 9784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:04.879046 9784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:04.879149 9784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:04.879410 9784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.3
I20260924 11:34:04.889142 9784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:35865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab
--principal=kudu/127.6.212.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.212.3
--webserver_port=0
--tserver_master_addrs=127.6.212.62:37475
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.3
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:34:04.890465 9784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:04.891769 9784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:04.905123 9792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:04.905045 9789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:04.905098 9790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:04.908910 9784 server_base.cc:1034] running on GCE node
I20260924 11:34:04.909549 9784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:04.910516 9784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:04.911737 9784 hybrid_clock.cc:648] HybridClock initialized: now 1790249644911686 us; error 48 us; skew 500 ppm
I20260924 11:34:04.912047 9784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:34:04 dist-test-slave-cq7c krb5kdc[9410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249644, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:04.915052 9784 init.cc:377] Logged in from keytab as kudu/127.6.212.3@KRBTEST.COM (short username kudu)
I20260924 11:34:04.916664 9784 webserver.cc:460] Webserver started at http://127.6.212.3:41523/ using document root <none> and password file <none>
I20260924 11:34:04.917372 9784 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:04.917516 9784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:04.917835 9784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:34:04.920768 9784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-2/data/instance:
uuid: "d5a7c73d7b504618b51ac70422482892"
format_stamp: "Formatted at 2026-09-24 11:34:04 on dist-test-slave-cq7c"
server_key: "14f2b50fc9b1f63a10e6e1b1c5a5ca92"
server_key_iv: "0764cf6c0f0a25fd5fe438530a116ca4"
server_key_version: "encryptionkey@0"
I20260924 11:34:04.921545 9784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance:
uuid: "d5a7c73d7b504618b51ac70422482892"
format_stamp: "Formatted at 2026-09-24 11:34:04 on dist-test-slave-cq7c"
server_key: "14f2b50fc9b1f63a10e6e1b1c5a5ca92"
server_key_iv: "0764cf6c0f0a25fd5fe438530a116ca4"
server_key_version: "encryptionkey@0"
I20260924 11:34:04.926806 9784 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 11:34:04.930555 9799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:04.932020 9784 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 11:34:04.932195 9784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-2/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-2/wal
uuid: "d5a7c73d7b504618b51ac70422482892"
format_stamp: "Formatted at 2026-09-24 11:34:04 on dist-test-slave-cq7c"
server_key: "14f2b50fc9b1f63a10e6e1b1c5a5ca92"
server_key_iv: "0764cf6c0f0a25fd5fe438530a116ca4"
server_key_version: "encryptionkey@0"
I20260924 11:34:04.932410 9784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:04.945652 9784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:04.949640 9784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:34:04.949992 9784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:04.950974 9784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:34:04.952450 9784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:34:04.952553 9784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:04.952705 9784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:34:04.952787 9784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:04.985841 9784 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.3:46493
I20260924 11:34:04.985886 9911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.3:46493 every 8 connection(s)
I20260924 11:34:04.987262 9784 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
I20260924 11:34:04.997174 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 9784
I20260924 11:34:04.997359 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance
I20260924 11:34:04.997870 6992 external_mini_cluster.cc:1428] Setting key 3ed89f25e39bdc103acccb9bef8fe0b8
Sep 24 11:34:04 dist-test-slave-cq7c krb5kdc[9410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249644, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.3@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:34:05.007891 9914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:04.989965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:59829 (local address 127.6.212.62:37475)
0924 11:34:04.990446 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:04.990452 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:04.991510 (+ 1058us) server_negotiation.cc:407] Connection header received
0924 11:34:04.993550 (+ 2040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:04.993553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:04.993619 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:04.993718 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:04.995178 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:04.996018 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:04.997703 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:04.998364 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:04.998501 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:05.002536 (+ 4035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:05.002541 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:05.002544 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:05.002576 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:05.005209 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:05.005869 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:05.005873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:05.005875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:05.005990 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:05.006365 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:05.006368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:05.006369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:05.006607 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:05.006710 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:05.007456 (+ 746us) server_negotiation.cc:299] Negotiation successful
0924 11:34:05.007621 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":217,"threads_started":1}
I20260924 11:34:05.008940 9915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:04.990475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37475 (local address 127.6.212.3:59829)
0924 11:34:04.991323 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:04.991389 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:04.993219 (+ 1830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:04.993921 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:04.993930 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:04.994479 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:04.995020 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:04.995037 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:04.996158 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:04.996161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:04.997567 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:04.997578 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:04.998598 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:04.998606 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:04.998890 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:04.999798 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:04.999828 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:05.002343 (+ 2515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:05.005348 (+ 3005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:05.005355 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:05.005368 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:05.005773 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:05.006081 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:05.006084 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:05.006087 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:05.006276 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:05.006714 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:05.006722 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:05.007039 (+ 317us) client_negotiation.cc:769] Sending connection context
0924 11:34:05.007319 (+ 280us) client_negotiation.cc:240] Negotiation successful
0924 11:34:05.007790 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":330,"threads_started":1}
I20260924 11:34:05.010439 9912 heartbeater.cc:347] Connected to a master server at 127.6.212.62:37475
I20260924 11:34:05.010722 9912 heartbeater.cc:464] Registering TS with master...
I20260924 11:34:05.011512 9912 heartbeater.cc:511] Master 127.6.212.62:37475 requested a full tablet report, sending...
I20260924 11:34:05.012957 9455 ts_manager.cc:194] Registered new tserver with Master: d5a7c73d7b504618b51ac70422482892 (127.6.212.3:46493)
I20260924 11:34:05.013733 6992 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:34:05.014040 9455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.3@KRBTEST.COM'} at 127.6.212.3:59829
I20260924 11:34:05.021992 9778 heartbeater.cc:503] Master 127.6.212.62:37475 was elected leader, sending a full tablet report...
W20260924 11:34:05.044528 9914 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:34:05.042040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42074 (local address 127.6.212.62:37475)
0924 11:34:05.042289 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:05.042297 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:05.042562 (+ 265us) server_negotiation.cc:407] Connection header received
0924 11:34:05.042723 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:05.042727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:05.042802 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:05.042888 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:05.044227 (+ 1339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42074: BlockingRecv error: recv got EOF from 127.0.0.1:42074 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 11:34:05.047410 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 9516
W20260924 11:34:05.050371 6992 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 9516: Not authorized: Client connection negotiation failed: client connection to 127.6.212.1:42705: server requires authentication, but client does not have Kerberos credentials available
I20260924 11:34:05.050577 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 9516
I20260924 11:34:05.067095 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 9650
W20260924 11:34:05.070160 6992 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 9650: Not authorized: Client connection negotiation failed: client connection to 127.6.212.2:40795: server requires authentication, but client does not have Kerberos credentials available
I20260924 11:34:05.070279 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 9650
I20260924 11:34:05.085947 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 9784
W20260924 11:34:05.089056 6992 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 9784: Not authorized: Client connection negotiation failed: client connection to 127.6.212.3:46493: server requires authentication, but client does not have Kerberos credentials available
I20260924 11:34:05.089244 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 9784
I20260924 11:34:05.104733 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 9423
I20260924 11:34:05.106012 9485 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:34:05.208791 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 9423
2026-09-24T11:34:05Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (6417 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790249617057452-6992-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:34:05 dist-test-slave-cq7c krb5kdc[9930](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:34:05 dist-test-slave-cq7c krb5kdc[9930](info): set up 2 sockets
Sep 24 11:34:05 dist-test-slave-cq7c krb5kdc[9930](info): commencing operation
krb5kdc: starting...
W20260924 11:34:06.327797 6992 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Sep 24 11:34:06 dist-test-slave-cq7c krb5kdc[9930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:34:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:34:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.212.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
I20260924 11:34:06.527964 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:40179
--webserver_interface=127.6.212.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:41417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:40179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:34:06.855434 9943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:06.855916 9943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:06.856066 9943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:06.863588 9943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:34:06.863696 9943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:06.863776 9943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:06.863862 9943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:34:06.863934 9943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:34:06.873677 9943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:41417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:40179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:40179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.212.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:34:06.874974 9943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:06.876340 9943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:06.888649 9948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:06.888805 9949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:06.888669 9951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:06.889464 9943 server_base.cc:1034] running on GCE node
I20260924 11:34:06.890210 9943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:06.891640 9943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:06.892843 9943 hybrid_clock.cc:648] HybridClock initialized: now 1790249646892791 us; error 41 us; skew 500 ppm
I20260924 11:34:06.893196 9943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:34:06 dist-test-slave-cq7c krb5kdc[9930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249646, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:06.896503 9943 init.cc:377] Logged in from keytab as kudu/127.6.212.62@KRBTEST.COM (short username kudu)
I20260924 11:34:06.897997 9943 webserver.cc:460] Webserver started at http://127.6.212.62:46459/ using document root <none> and password file <none>
I20260924 11:34:06.898663 9943 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:06.898764 9943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:06.899017 9943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:34:06.901988 9943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "02cc3698b81d435792f451893e6f6ee6"
format_stamp: "Formatted at 2026-09-24 11:34:06 on dist-test-slave-cq7c"
server_key: "d0182a5e51ea076954a7df2023c1e16c"
server_key_iv: "52f7287c34932be4053e397749cea7f9"
server_key_version: "encryptionkey@0"
I20260924 11:34:06.902753 9943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "02cc3698b81d435792f451893e6f6ee6"
format_stamp: "Formatted at 2026-09-24 11:34:06 on dist-test-slave-cq7c"
server_key: "d0182a5e51ea076954a7df2023c1e16c"
server_key_iv: "52f7287c34932be4053e397749cea7f9"
server_key_version: "encryptionkey@0"
I20260924 11:34:06.908725 9943 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260924 11:34:06.912103 9958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:06.913492 9943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:34:06.913795 9943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "02cc3698b81d435792f451893e6f6ee6"
format_stamp: "Formatted at 2026-09-24 11:34:06 on dist-test-slave-cq7c"
server_key: "d0182a5e51ea076954a7df2023c1e16c"
server_key_iv: "52f7287c34932be4053e397749cea7f9"
server_key_version: "encryptionkey@0"
I20260924 11:34:06.914026 9943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:06.926105 9943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:06.930699 9943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:34:06.931020 9943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:06.950865 9943 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.62:40179
I20260924 11:34:06.950953 10009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.62:40179 every 8 connection(s)
I20260924 11:34:06.952487 9943 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:34:06.956399 10010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:34:06.957660 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 9943
I20260924 11:34:06.957890 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:34:06.958694 6992 external_mini_cluster.cc:1428] Setting key fa3200747bc02d437e8df50a09ebcb46
I20260924 11:34:06.972348 10010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02cc3698b81d435792f451893e6f6ee6: Bootstrap starting.
Sep 24 11:34:06 dist-test-slave-cq7c krb5kdc[9930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:34:06.976210 10010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02cc3698b81d435792f451893e6f6ee6: Neither blocks nor log segments found. Creating new log.
I20260924 11:34:06.977468 10010 log.cc:824] T 00000000000000000000000000000000 P 02cc3698b81d435792f451893e6f6ee6: Log is configured to *not* fsync() on all Append() calls
I20260924 11:34:06.980683 10010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02cc3698b81d435792f451893e6f6ee6: No bootstrap required, opened a new log
I20260924 11:34:06.982987 10013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:06.960632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49736 (local address 127.6.212.62:40179)
0924 11:34:06.961444 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:06.961455 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:06.961496 (+ 41us) server_negotiation.cc:407] Connection header received
0924 11:34:06.963027 (+ 1531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:06.963047 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:06.963508 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:06.963834 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:06.966038 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:06.967271 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:06.969349 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:06.970036 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:06.970166 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:06.975231 (+ 5065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:06.975238 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:06.975251 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:06.975295 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:06.978573 (+ 3278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:06.979240 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:06.979246 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:06.979251 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:06.979471 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:06.979844 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:06.979850 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:06.979854 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:06.980373 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:06.980672 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:06.981121 (+ 449us) server_negotiation.cc:299] Negotiation successful
0924 11:34:06.981602 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":369,"threads_started":1}
I20260924 11:34:06.987126 6992 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:34:06.992547 10010 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02cc3698b81d435792f451893e6f6ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02cc3698b81d435792f451893e6f6ee6" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 40179 } }
I20260924 11:34:06.993010 10010 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02cc3698b81d435792f451893e6f6ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:34:06.993130 10010 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02cc3698b81d435792f451893e6f6ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02cc3698b81d435792f451893e6f6ee6, State: Initialized, Role: FOLLOWER
I20260924 11:34:06.993804 10010 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02cc3698b81d435792f451893e6f6ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02cc3698b81d435792f451893e6f6ee6" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 40179 } }
I20260924 11:34:06.994102 10010 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02cc3698b81d435792f451893e6f6ee6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:34:06.994254 10010 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02cc3698b81d435792f451893e6f6ee6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:34:06.994428 10010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02cc3698b81d435792f451893e6f6ee6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:34:06.997123 10010 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02cc3698b81d435792f451893e6f6ee6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02cc3698b81d435792f451893e6f6ee6" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 40179 } }
I20260924 11:34:06.997615 10010 leader_election.cc:302] T 00000000000000000000000000000000 P 02cc3698b81d435792f451893e6f6ee6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02cc3698b81d435792f451893e6f6ee6; no voters:
I20260924 11:34:06.998209 10010 leader_election.cc:288] T 00000000000000000000000000000000 P 02cc3698b81d435792f451893e6f6ee6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:34:06.998332 10016 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02cc3698b81d435792f451893e6f6ee6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:34:06.999626 10016 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02cc3698b81d435792f451893e6f6ee6 [term 1 LEADER]: Becoming Leader. State: Replica: 02cc3698b81d435792f451893e6f6ee6, State: Running, Role: LEADER
I20260924 11:34:07.000162 10016 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02cc3698b81d435792f451893e6f6ee6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02cc3698b81d435792f451893e6f6ee6" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 40179 } }
I20260924 11:34:07.000634 10010 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02cc3698b81d435792f451893e6f6ee6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:34:07.002882 10017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02cc3698b81d435792f451893e6f6ee6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02cc3698b81d435792f451893e6f6ee6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02cc3698b81d435792f451893e6f6ee6" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 40179 } } }
I20260924 11:34:07.003216 10017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02cc3698b81d435792f451893e6f6ee6 [sys.catalog]: This master's current role is: LEADER
I20260924 11:34:07.005887 10018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02cc3698b81d435792f451893e6f6ee6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02cc3698b81d435792f451893e6f6ee6. Latest consensus state: current_term: 1 leader_uuid: "02cc3698b81d435792f451893e6f6ee6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02cc3698b81d435792f451893e6f6ee6" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 40179 } } }
I20260924 11:34:07.006359 10018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02cc3698b81d435792f451893e6f6ee6 [sys.catalog]: This master's current role is: LEADER
I20260924 11:34:07.010470 10021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:34:07.016309 10021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:34:07.028734 10021 catalog_manager.cc:1347] Generated new cluster ID: 77fa9048dde64b5b856cdbd6db9e5138
I20260924 11:34:07.028890 10021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:34:07.034847 10037 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 11:34:07.047264 10013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:07.037962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49746 (local address 127.6.212.62:40179)
0924 11:34:07.038119 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:07.038125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:07.038695 (+ 570us) server_negotiation.cc:407] Connection header received
0924 11:34:07.038880 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:07.038887 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:07.038976 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:07.039106 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:07.040246 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:07.041529 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:07.043299 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:07.044106 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:07.044269 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:07.045449 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:07.045457 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:07.045463 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:34:07.046423 (+ 960us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:07.046506 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:34:07.046540 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:07.046654 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:07.046787 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 11:34:07.046886 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 11:34:07.062074 10021 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:34:07.063424 10021 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:34:07.069720 10021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02cc3698b81d435792f451893e6f6ee6: Generated new TSK 0
I20260924 11:34:07.070605 10021 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:34:07.087433 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 9943
I20260924 11:34:07.088599 10005 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:34:07.180152 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 9943
2026-09-24T11:34:07Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1966 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:34:07 dist-test-slave-cq7c krb5kdc[10044](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:34:07 dist-test-slave-cq7c krb5kdc[10044](info): set up 2 sockets
Sep 24 11:34:07 dist-test-slave-cq7c krb5kdc[10044](info): commencing operation
krb5kdc: starting...
W20260924 11:34:08.263942 6992 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.005s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Sep 24 11:34:08 dist-test-slave-cq7c krb5kdc[10044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:34:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:34:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.212.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
I20260924 11:34:08.460106 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:43371
--webserver_interface=127.6.212.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:38257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:43371
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:34:08.776613 10057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:08.776947 10057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:08.777107 10057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:08.784206 10057 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:34:08.784348 10057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:08.784451 10057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:08.784554 10057 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:34:08.784637 10057 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:34:08.794317 10057 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:38257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:43371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:43371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.212.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:34:08.795655 10057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:08.796977 10057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:08.808837 10065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:08.808933 10063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:08.808944 10062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:08.809983 10057 server_base.cc:1034] running on GCE node
I20260924 11:34:08.810708 10057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:08.812183 10057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:08.813388 10057 hybrid_clock.cc:648] HybridClock initialized: now 1790249648813345 us; error 39 us; skew 500 ppm
I20260924 11:34:08.813722 10057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:34:08 dist-test-slave-cq7c krb5kdc[10044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249648, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:08.816972 10057 init.cc:377] Logged in from keytab as kudu/127.6.212.62@KRBTEST.COM (short username kudu)
I20260924 11:34:08.818437 10057 webserver.cc:460] Webserver started at http://127.6.212.62:37637/ using document root <none> and password file <none>
I20260924 11:34:08.819113 10057 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:08.819259 10057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:08.819613 10057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:34:08.822638 10057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "261c8a5beae44cc4b3b3756debeacdae"
format_stamp: "Formatted at 2026-09-24 11:34:08 on dist-test-slave-cq7c"
server_key: "c61b11f63a689562e36dcbe27475bee0"
server_key_iv: "f5628192d0c0982f52e9679af94a61aa"
server_key_version: "encryptionkey@0"
I20260924 11:34:08.823483 10057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "261c8a5beae44cc4b3b3756debeacdae"
format_stamp: "Formatted at 2026-09-24 11:34:08 on dist-test-slave-cq7c"
server_key: "c61b11f63a689562e36dcbe27475bee0"
server_key_iv: "f5628192d0c0982f52e9679af94a61aa"
server_key_version: "encryptionkey@0"
I20260924 11:34:08.828670 10057 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 11:34:08.832376 10072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:08.834117 10057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 11:34:08.834316 10057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "261c8a5beae44cc4b3b3756debeacdae"
format_stamp: "Formatted at 2026-09-24 11:34:08 on dist-test-slave-cq7c"
server_key: "c61b11f63a689562e36dcbe27475bee0"
server_key_iv: "f5628192d0c0982f52e9679af94a61aa"
server_key_version: "encryptionkey@0"
I20260924 11:34:08.834528 10057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:08.853271 10057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:08.864532 10057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:34:08.864868 10057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:08.883930 10057 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.62:43371
I20260924 11:34:08.883972 10123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.62:43371 every 8 connection(s)
I20260924 11:34:08.885293 10057 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:34:08.888185 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 10057
I20260924 11:34:08.888446 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:34:08.889294 6992 external_mini_cluster.cc:1428] Setting key ec313bdc1042bf48c947e1c85e5f94ca
I20260924 11:34:08.890015 10124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:34:08.902964 10124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 261c8a5beae44cc4b3b3756debeacdae: Bootstrap starting.
Sep 24 11:34:08 dist-test-slave-cq7c krb5kdc[10044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:34:08.906494 10124 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 261c8a5beae44cc4b3b3756debeacdae: Neither blocks nor log segments found. Creating new log.
I20260924 11:34:08.907691 10124 log.cc:824] T 00000000000000000000000000000000 P 261c8a5beae44cc4b3b3756debeacdae: Log is configured to *not* fsync() on all Append() calls
I20260924 11:34:08.910712 10124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 261c8a5beae44cc4b3b3756debeacdae: No bootstrap required, opened a new log
I20260924 11:34:08.911536 10127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:08.891571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44488 (local address 127.6.212.62:43371)
0924 11:34:08.892322 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:08.892333 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:08.892372 (+ 39us) server_negotiation.cc:407] Connection header received
0924 11:34:08.894723 (+ 2351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:08.894744 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:08.895122 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:08.895572 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:08.896929 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:08.898186 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:08.899421 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:08.900154 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:08.900381 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:08.904434 (+ 4053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:08.904442 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:08.904455 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:08.904495 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:08.907498 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:08.908252 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:08.908262 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:08.908268 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:08.908433 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:08.908833 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:08.908845 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:08.908847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:08.909316 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:08.909623 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:08.910042 (+ 419us) server_negotiation.cc:299] Negotiation successful
0924 11:34:08.910440 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":336,"threads_started":1}
I20260924 11:34:08.919318 10124 raft_consensus.cc:348] T 00000000000000000000000000000000 P 261c8a5beae44cc4b3b3756debeacdae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261c8a5beae44cc4b3b3756debeacdae" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43371 } }
I20260924 11:34:08.919795 10124 raft_consensus.cc:374] T 00000000000000000000000000000000 P 261c8a5beae44cc4b3b3756debeacdae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:34:08.919919 10124 raft_consensus.cc:729] T 00000000000000000000000000000000 P 261c8a5beae44cc4b3b3756debeacdae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 261c8a5beae44cc4b3b3756debeacdae, State: Initialized, Role: FOLLOWER
I20260924 11:34:08.920499 10124 consensus_queue.cc:260] T 00000000000000000000000000000000 P 261c8a5beae44cc4b3b3756debeacdae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261c8a5beae44cc4b3b3756debeacdae" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43371 } }
I20260924 11:34:08.920753 10124 raft_consensus.cc:388] T 00000000000000000000000000000000 P 261c8a5beae44cc4b3b3756debeacdae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:34:08.920908 10124 raft_consensus.cc:482] T 00000000000000000000000000000000 P 261c8a5beae44cc4b3b3756debeacdae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:34:08.921094 10124 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 261c8a5beae44cc4b3b3756debeacdae [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:34:08.923907 10124 raft_consensus.cc:504] T 00000000000000000000000000000000 P 261c8a5beae44cc4b3b3756debeacdae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261c8a5beae44cc4b3b3756debeacdae" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43371 } }
I20260924 11:34:08.924427 10124 leader_election.cc:302] T 00000000000000000000000000000000 P 261c8a5beae44cc4b3b3756debeacdae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 261c8a5beae44cc4b3b3756debeacdae; no voters:
I20260924 11:34:08.925104 10124 leader_election.cc:288] T 00000000000000000000000000000000 P 261c8a5beae44cc4b3b3756debeacdae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:34:08.925251 10129 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 261c8a5beae44cc4b3b3756debeacdae [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:34:08.926517 10129 raft_consensus.cc:686] T 00000000000000000000000000000000 P 261c8a5beae44cc4b3b3756debeacdae [term 1 LEADER]: Becoming Leader. State: Replica: 261c8a5beae44cc4b3b3756debeacdae, State: Running, Role: LEADER
I20260924 11:34:08.927138 10129 consensus_queue.cc:237] T 00000000000000000000000000000000 P 261c8a5beae44cc4b3b3756debeacdae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261c8a5beae44cc4b3b3756debeacdae" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43371 } }
I20260924 11:34:08.927896 10124 sys_catalog.cc:564] T 00000000000000000000000000000000 P 261c8a5beae44cc4b3b3756debeacdae [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:34:08.929927 10131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 261c8a5beae44cc4b3b3756debeacdae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 261c8a5beae44cc4b3b3756debeacdae. Latest consensus state: current_term: 1 leader_uuid: "261c8a5beae44cc4b3b3756debeacdae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261c8a5beae44cc4b3b3756debeacdae" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43371 } } }
I20260924 11:34:08.929940 10130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 261c8a5beae44cc4b3b3756debeacdae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "261c8a5beae44cc4b3b3756debeacdae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261c8a5beae44cc4b3b3756debeacdae" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43371 } } }
I20260924 11:34:08.930328 10131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 261c8a5beae44cc4b3b3756debeacdae [sys.catalog]: This master's current role is: LEADER
I20260924 11:34:08.930428 10130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 261c8a5beae44cc4b3b3756debeacdae [sys.catalog]: This master's current role is: LEADER
I20260924 11:34:08.931957 10134 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:34:08.937326 10134 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:34:08.946659 10134 catalog_manager.cc:1347] Generated new cluster ID: c581feb68f3c4be790c9d04967b76df2
I20260924 11:34:08.946813 10134 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:34:08.957468 10134 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:34:08.958902 10134 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:34:08.969183 10134 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 261c8a5beae44cc4b3b3756debeacdae: Generated new TSK 0
I20260924 11:34:08.970080 10134 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.212.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab.
Entry for principal kudu/127.6.212.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab.
I20260924 11:34:09.027685 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.1:0
--local_ip_for_outbound_sockets=127.6.212.1
--webserver_interface=127.6.212.1
--webserver_port=0
--tserver_master_addrs=127.6.212.62:43371
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab
--principal=kudu/127.6.212.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:38257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:34:09.386130 10150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:09.386407 10150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:09.386502 10150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:09.393730 10150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:09.393836 10150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:09.393944 10150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.1
I20260924 11:34:09.403334 10150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:38257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab
--principal=kudu/127.6.212.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.212.1
--webserver_port=0
--tserver_master_addrs=127.6.212.62:43371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.1
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:34:09.404695 10150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:09.405929 10150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:09.418581 10155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:09.418731 10156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:09.419075 10158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:09.419950 10150 server_base.cc:1034] running on GCE node
I20260924 11:34:09.420570 10150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:09.421535 10150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:09.422739 10150 hybrid_clock.cc:648] HybridClock initialized: now 1790249649422691 us; error 41 us; skew 500 ppm
I20260924 11:34:09.423059 10150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:34:09 dist-test-slave-cq7c krb5kdc[10044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249649, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:09.426462 10150 init.cc:377] Logged in from keytab as kudu/127.6.212.1@KRBTEST.COM (short username kudu)
I20260924 11:34:09.428000 10150 webserver.cc:460] Webserver started at http://127.6.212.1:39293/ using document root <none> and password file <none>
I20260924 11:34:09.428711 10150 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:09.428854 10150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:09.429205 10150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:34:09.432025 10150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-0/data/instance:
uuid: "29c300c27e0646c4a72a21a1feb15869"
format_stamp: "Formatted at 2026-09-24 11:34:09 on dist-test-slave-cq7c"
server_key: "5186e2d96807ce7b1bbbda21970da9cc"
server_key_iv: "17b04783ebc48068dd4c00cb5bc0c0d7"
server_key_version: "encryptionkey@0"
I20260924 11:34:09.432870 10150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance:
uuid: "29c300c27e0646c4a72a21a1feb15869"
format_stamp: "Formatted at 2026-09-24 11:34:09 on dist-test-slave-cq7c"
server_key: "5186e2d96807ce7b1bbbda21970da9cc"
server_key_iv: "17b04783ebc48068dd4c00cb5bc0c0d7"
server_key_version: "encryptionkey@0"
I20260924 11:34:09.437947 10150 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 11:34:09.441349 10165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:09.442636 10150 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 11:34:09.442817 10150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-0/wal
uuid: "29c300c27e0646c4a72a21a1feb15869"
format_stamp: "Formatted at 2026-09-24 11:34:09 on dist-test-slave-cq7c"
server_key: "5186e2d96807ce7b1bbbda21970da9cc"
server_key_iv: "17b04783ebc48068dd4c00cb5bc0c0d7"
server_key_version: "encryptionkey@0"
I20260924 11:34:09.442999 10150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:09.484799 10150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:09.489215 10150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:34:09.489516 10150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:09.490450 10150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:34:09.491868 10150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:34:09.491971 10150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:09.492112 10150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:34:09.492187 10150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:09.524745 10150 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.1:37703
I20260924 11:34:09.524786 10277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.1:37703 every 8 connection(s)
I20260924 11:34:09.526018 10150 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
I20260924 11:34:09.526975 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 10150
I20260924 11:34:09.527205 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance
I20260924 11:34:09.527947 6992 external_mini_cluster.cc:1428] Setting key 7bacc8f3422de4513191f00bbd2783e6
Sep 24 11:34:09 dist-test-slave-cq7c krb5kdc[10044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249649, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.212.2@KRBTEST.COM; defaulting to no policy
I20260924 11:34:09.547915 10280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:09.528662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:54495 (local address 127.6.212.62:43371)
0924 11:34:09.529122 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:09.529128 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:09.530329 (+ 1201us) server_negotiation.cc:407] Connection header received
0924 11:34:09.532176 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:09.532181 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:09.532261 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:09.532385 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:09.533808 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:09.534499 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:09.536674 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:09.537068 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:09.537198 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:09.541158 (+ 3960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:09.541167 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:09.541176 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:09.541219 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:09.544866 (+ 3647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:09.545534 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:09.545540 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:09.545546 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:09.545702 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:09.546102 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:09.546108 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:09.546112 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:09.546471 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:09.546672 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:09.547496 (+ 824us) server_negotiation.cc:299] Negotiation successful
0924 11:34:09.547660 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":211,"threads_started":1}
I20260924 11:34:09.549043 10281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:09.529330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:43371 (local address 127.6.212.1:54495)
0924 11:34:09.530166 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:09.530226 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:09.531920 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:09.532613 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:09.532621 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:09.533095 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:09.533642 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:09.533659 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:09.534651 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:09.534656 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:09.536533 (+ 1877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:09.536544 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:09.537299 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:09.537306 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:09.537545 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:09.538224 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:09.538274 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:09.540941 (+ 2667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:09.544992 (+ 4051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:09.544998 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:09.545010 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:09.545398 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:09.545814 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:09.545817 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:09.545819 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:09.545986 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:09.546610 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:09.546618 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:09.546984 (+ 366us) client_negotiation.cc:769] Sending connection context
0924 11:34:09.547349 (+ 365us) client_negotiation.cc:240] Negotiation successful
0924 11:34:09.547879 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":302,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.2@KRBTEST.COM" created.
I20260924 11:34:09.550658 10278 heartbeater.cc:347] Connected to a master server at 127.6.212.62:43371
I20260924 11:34:09.550993 10278 heartbeater.cc:464] Registering TS with master...
I20260924 11:34:09.551923 10278 heartbeater.cc:511] Master 127.6.212.62:43371 requested a full tablet report, sending...
I20260924 11:34:09.554239 10089 ts_manager.cc:194] Registered new tserver with Master: 29c300c27e0646c4a72a21a1feb15869 (127.6.212.1:37703)
I20260924 11:34:09.555979 10089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.1@KRBTEST.COM'} at 127.6.212.1:54495
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab.
Entry for principal kudu/127.6.212.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab.
I20260924 11:34:09.574704 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.2:0
--local_ip_for_outbound_sockets=127.6.212.2
--webserver_interface=127.6.212.2
--webserver_port=0
--tserver_master_addrs=127.6.212.62:43371
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab
--principal=kudu/127.6.212.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:38257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:34:09.901669 10284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:09.902040 10284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:09.902180 10284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:09.911450 10284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:09.911553 10284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:09.911677 10284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.2
I20260924 11:34:09.921414 10284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:38257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab
--principal=kudu/127.6.212.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.212.2
--webserver_port=0
--tserver_master_addrs=127.6.212.62:43371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.2
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:34:09.922650 10284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:09.923902 10284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:09.936846 10292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:09.936820 10289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:09.936887 10290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:09.938449 10284 server_base.cc:1034] running on GCE node
I20260924 11:34:09.939148 10284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:09.940269 10284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:09.941501 10284 hybrid_clock.cc:648] HybridClock initialized: now 1790249649941455 us; error 50 us; skew 500 ppm
I20260924 11:34:09.941843 10284 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:34:09 dist-test-slave-cq7c krb5kdc[10044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249649, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:09.945165 10284 init.cc:377] Logged in from keytab as kudu/127.6.212.2@KRBTEST.COM (short username kudu)
I20260924 11:34:09.946652 10284 webserver.cc:460] Webserver started at http://127.6.212.2:40079/ using document root <none> and password file <none>
I20260924 11:34:09.947458 10284 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:09.947598 10284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:09.947906 10284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:34:09.950960 10284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-1/data/instance:
uuid: "6be49e1b00154028a5daea8f44b72ead"
format_stamp: "Formatted at 2026-09-24 11:34:09 on dist-test-slave-cq7c"
server_key: "7cb894cf8e59db2716e10e27165ba903"
server_key_iv: "3c640f4a6090714c3a19e8b7486e5f46"
server_key_version: "encryptionkey@0"
I20260924 11:34:09.951812 10284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance:
uuid: "6be49e1b00154028a5daea8f44b72ead"
format_stamp: "Formatted at 2026-09-24 11:34:09 on dist-test-slave-cq7c"
server_key: "7cb894cf8e59db2716e10e27165ba903"
server_key_iv: "3c640f4a6090714c3a19e8b7486e5f46"
server_key_version: "encryptionkey@0"
I20260924 11:34:09.956979 10284 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 11:34:09.960707 10299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:09.962085 10284 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:34:09.962265 10284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-1/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-1/wal
uuid: "6be49e1b00154028a5daea8f44b72ead"
format_stamp: "Formatted at 2026-09-24 11:34:09 on dist-test-slave-cq7c"
server_key: "7cb894cf8e59db2716e10e27165ba903"
server_key_iv: "3c640f4a6090714c3a19e8b7486e5f46"
server_key_version: "encryptionkey@0"
I20260924 11:34:09.962467 10284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:09.998097 10284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:10.002239 10284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:34:10.002576 10284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:10.003619 10284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:34:10.005023 10284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:34:10.005137 10284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:10.005300 10284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:34:10.005374 10284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:10.039748 10284 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.2:36347
I20260924 11:34:10.039796 10411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.2:36347 every 8 connection(s)
I20260924 11:34:10.041030 10284 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
I20260924 11:34:10.042519 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 10284
I20260924 11:34:10.042703 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance
I20260924 11:34:10.043100 6992 external_mini_cluster.cc:1428] Setting key 5692bee5a473f10d3ccb240d3c718329
Sep 24 11:34:10 dist-test-slave-cq7c krb5kdc[10044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249649, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.2@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.212.3@KRBTEST.COM; defaulting to no policy
I20260924 11:34:10.063077 10280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:10.043781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:58189 (local address 127.6.212.62:43371)
0924 11:34:10.043985 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:10.043991 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:10.045263 (+ 1272us) server_negotiation.cc:407] Connection header received
0924 11:34:10.047052 (+ 1789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:10.047057 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:10.047142 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:10.047246 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:10.048691 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:10.049377 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:10.051799 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:10.052230 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:10.052330 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:10.057425 (+ 5095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:10.057429 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:10.057433 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:10.057461 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:10.060029 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:10.060902 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:10.060905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:10.060907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:10.061023 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:10.061484 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:10.061487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:10.061489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:10.061730 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:10.061842 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:10.062690 (+ 848us) server_negotiation.cc:299] Negotiation successful
0924 11:34:10.062847 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 11:34:10.064235 10414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:10.044333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:43371 (local address 127.6.212.2:58189)
0924 11:34:10.045098 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:10.045151 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:10.046804 (+ 1653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:10.047505 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:10.047513 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:10.047988 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:10.048531 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:10.048547 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:10.049538 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:10.049544 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:10.051652 (+ 2108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:10.051670 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:10.052451 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:10.052461 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:10.052779 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:10.053580 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:10.053621 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:10.057210 (+ 3589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:10.060167 (+ 2957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:10.060178 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:10.060197 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:10.060785 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:10.061130 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:10.061135 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:10.061138 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:10.061374 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:10.061865 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:10.061877 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:10.062294 (+ 417us) client_negotiation.cc:769] Sending connection context
0924 11:34:10.062616 (+ 322us) client_negotiation.cc:240] Negotiation successful
0924 11:34:10.063012 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":302,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.3@KRBTEST.COM" created.
I20260924 11:34:10.065820 10412 heartbeater.cc:347] Connected to a master server at 127.6.212.62:43371
I20260924 11:34:10.066190 10412 heartbeater.cc:464] Registering TS with master...
I20260924 11:34:10.067054 10412 heartbeater.cc:511] Master 127.6.212.62:43371 requested a full tablet report, sending...
I20260924 11:34:10.068598 10089 ts_manager.cc:194] Registered new tserver with Master: 6be49e1b00154028a5daea8f44b72ead (127.6.212.2:36347)
I20260924 11:34:10.069379 10089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.2@KRBTEST.COM'} at 127.6.212.2:58189
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab.
Entry for principal kudu/127.6.212.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab.
I20260924 11:34:10.090253 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.3:0
--local_ip_for_outbound_sockets=127.6.212.3
--webserver_interface=127.6.212.3
--webserver_port=0
--tserver_master_addrs=127.6.212.62:43371
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab
--principal=kudu/127.6.212.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:38257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:34:10.411309 10417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:10.411653 10417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:10.411798 10417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:10.419154 10417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:10.419255 10417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:10.419476 10417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.3
I20260924 11:34:10.429399 10417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:38257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab
--principal=kudu/127.6.212.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.212.3
--webserver_port=0
--tserver_master_addrs=127.6.212.62:43371
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.3
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:34:10.430730 10417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:10.432055 10417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:10.445128 10423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:10.445207 10425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:10.445212 10422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:10.446115 10417 server_base.cc:1034] running on GCE node
I20260924 11:34:10.446693 10417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:10.447737 10417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:10.448947 10417 hybrid_clock.cc:648] HybridClock initialized: now 1790249650448890 us; error 46 us; skew 500 ppm
I20260924 11:34:10.449259 10417 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:34:10 dist-test-slave-cq7c krb5kdc[10044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249650, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:10.452333 10417 init.cc:377] Logged in from keytab as kudu/127.6.212.3@KRBTEST.COM (short username kudu)
I20260924 11:34:10.453758 10417 webserver.cc:460] Webserver started at http://127.6.212.3:37289/ using document root <none> and password file <none>
I20260924 11:34:10.454423 10417 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:10.454557 10417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:10.454849 10417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:34:10.457675 10417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-2/data/instance:
uuid: "98ed1e71b8eb43d2b171f8e778b7857b"
format_stamp: "Formatted at 2026-09-24 11:34:10 on dist-test-slave-cq7c"
server_key: "6933f0464abd83a75962e08cf845d4d3"
server_key_iv: "6a6c7f27c0fc99838b6800dcec6db3e6"
server_key_version: "encryptionkey@0"
I20260924 11:34:10.458418 10417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance:
uuid: "98ed1e71b8eb43d2b171f8e778b7857b"
format_stamp: "Formatted at 2026-09-24 11:34:10 on dist-test-slave-cq7c"
server_key: "6933f0464abd83a75962e08cf845d4d3"
server_key_iv: "6a6c7f27c0fc99838b6800dcec6db3e6"
server_key_version: "encryptionkey@0"
I20260924 11:34:10.463532 10417 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260924 11:34:10.466861 10432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:10.468250 10417 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:34:10.468427 10417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-2/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-2/wal
uuid: "98ed1e71b8eb43d2b171f8e778b7857b"
format_stamp: "Formatted at 2026-09-24 11:34:10 on dist-test-slave-cq7c"
server_key: "6933f0464abd83a75962e08cf845d4d3"
server_key_iv: "6a6c7f27c0fc99838b6800dcec6db3e6"
server_key_version: "encryptionkey@0"
I20260924 11:34:10.468623 10417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:10.483206 10417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:10.487074 10417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:34:10.487397 10417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:10.488327 10417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:34:10.489617 10417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:34:10.489696 10417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:10.489856 10417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:34:10.489907 10417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:10.520750 10417 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.3:43707
I20260924 11:34:10.520802 10544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.3:43707 every 8 connection(s)
I20260924 11:34:10.522009 10417 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
I20260924 11:34:10.526878 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 10417
I20260924 11:34:10.527087 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance
I20260924 11:34:10.527626 6992 external_mini_cluster.cc:1428] Setting key 4319da6c6097a98d7348caa6d26ffef9
Sep 24 11:34:10 dist-test-slave-cq7c krb5kdc[10044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249650, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.3@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:34:10.542375 10280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:10.524568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:52409 (local address 127.6.212.62:43371)
0924 11:34:10.524737 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:10.524741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:10.526102 (+ 1361us) server_negotiation.cc:407] Connection header received
0924 11:34:10.528738 (+ 2636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:10.528747 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:10.528833 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:10.528938 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:10.530862 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:10.531647 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:10.533263 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:10.533646 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:10.533767 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:10.537049 (+ 3282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:10.537053 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:10.537057 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:10.537086 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:10.539590 (+ 2504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:10.540227 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:10.540230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:10.540232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:10.540349 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:10.540695 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:10.540698 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:10.540700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:10.540944 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:10.541046 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:10.541792 (+ 746us) server_negotiation.cc:299] Negotiation successful
0924 11:34:10.542040 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 11:34:10.543517 10547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:10.525181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:43371 (local address 127.6.212.3:52409)
0924 11:34:10.525947 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:10.525983 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:10.528412 (+ 2429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:10.529311 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:10.529323 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:10.529925 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:10.530694 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:10.530713 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:10.531776 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:10.531780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:10.533100 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:10.533111 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:10.533854 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:10.533860 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:10.534048 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:10.534736 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:10.534757 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:10.536868 (+ 2111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:10.539701 (+ 2833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:10.539708 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:10.539720 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:10.540107 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:10.540434 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:10.540437 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:10.540440 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:10.540605 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:10.541058 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:10.541064 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:10.541467 (+ 403us) client_negotiation.cc:769] Sending connection context
0924 11:34:10.541737 (+ 270us) client_negotiation.cc:240] Negotiation successful
0924 11:34:10.542215 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":311,"threads_started":1}
I20260924 11:34:10.544734 10545 heartbeater.cc:347] Connected to a master server at 127.6.212.62:43371
I20260924 11:34:10.545014 10545 heartbeater.cc:464] Registering TS with master...
I20260924 11:34:10.545722 10545 heartbeater.cc:511] Master 127.6.212.62:43371 requested a full tablet report, sending...
I20260924 11:34:10.547160 10089 ts_manager.cc:194] Registered new tserver with Master: 98ed1e71b8eb43d2b171f8e778b7857b (127.6.212.3:43707)
I20260924 11:34:10.547995 10089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.3@KRBTEST.COM'} at 127.6.212.3:52409
I20260924 11:34:10.555052 6992 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:34:10.558832 10278 heartbeater.cc:503] Master 127.6.212.62:43371 was elected leader, sending a full tablet report...
Sep 24 11:34:10 dist-test-slave-cq7c krb5kdc[10044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249650, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 11:34:10 dist-test-slave-cq7c krb5kdc[10044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249650, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:34:10.602166 10280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:10.589296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44494 (local address 127.6.212.62:43371)
0924 11:34:10.589468 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:10.589473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:10.589886 (+ 413us) server_negotiation.cc:407] Connection header received
0924 11:34:10.590035 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:10.590039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:10.590100 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:10.590198 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:10.591570 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:10.592310 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:10.593484 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:10.593874 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:10.593969 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:10.597580 (+ 3611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:10.597584 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:10.597587 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:10.597620 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:10.599971 (+ 2351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:10.600559 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:10.600563 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:10.600565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:10.600681 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:10.600996 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:10.600999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:10.601001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:10.601287 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:10.601462 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:10.601704 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 11:34:10.601864 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260924 11:34:10.602806 10089 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44494
I20260924 11:34:10.604550 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 10150
Sep 24 11:34:10 dist-test-slave-cq7c krb5kdc[10044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249650, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.212.1@KRBTEST.COM
I20260924 11:34:10.619700 10556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:10.605489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56500 (local address 127.6.212.1:37703)
0924 11:34:10.606061 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:10.606068 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:10.606093 (+ 25us) server_negotiation.cc:407] Connection header received
0924 11:34:10.606155 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:10.606161 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:10.606375 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:10.606501 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:10.607820 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:10.608850 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:10.610028 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:10.610578 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:10.610674 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:10.614359 (+ 3685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:10.614371 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:10.614384 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:10.614414 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:10.617238 (+ 2824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:10.617849 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:10.617857 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:10.617860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:10.617993 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:10.618322 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:10.618327 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:10.618330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:10.618689 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:10.618925 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:10.619128 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 11:34:10.619330 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":274,"threads_started":1}
W20260924 11:34:10.621004 10273 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:56500
W20260924 11:34:10.622144 6992 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 10150: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:34:10.622351 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 10150
I20260924 11:34:10.638814 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 10284
Sep 24 11:34:10 dist-test-slave-cq7c krb5kdc[10044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249650, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.212.2@KRBTEST.COM
I20260924 11:34:10.653990 10557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:10.639716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53424 (local address 127.6.212.2:36347)
0924 11:34:10.640231 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:10.640238 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:10.640263 (+ 25us) server_negotiation.cc:407] Connection header received
0924 11:34:10.640322 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:10.640329 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:10.640535 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:10.640656 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:10.641754 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:10.642704 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:10.643941 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:10.644466 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:10.644559 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:10.648451 (+ 3892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:10.648462 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:10.648475 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:10.648505 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:10.651211 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:10.651972 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:10.651979 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:10.651982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:10.652095 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:10.652455 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:10.652466 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:10.652468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:10.652776 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:10.653090 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:10.653402 (+ 312us) server_negotiation.cc:299] Negotiation successful
0924 11:34:10.653652 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":249,"threads_started":1}
W20260924 11:34:10.655202 10407 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53424
W20260924 11:34:10.656283 6992 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 10284: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:34:10.656446 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 10284
I20260924 11:34:10.671950 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 10417
Sep 24 11:34:10 dist-test-slave-cq7c krb5kdc[10044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249650, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.212.3@KRBTEST.COM
I20260924 11:34:10.688054 10558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:10.673004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49384 (local address 127.6.212.3:43707)
0924 11:34:10.673627 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:10.673634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:10.673690 (+ 56us) server_negotiation.cc:407] Connection header received
0924 11:34:10.673746 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:10.673752 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:10.673955 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:10.674085 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:10.675255 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:10.676324 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:10.677482 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:10.678031 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:10.678126 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:10.682311 (+ 4185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:10.682320 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:10.682334 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:10.682364 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:10.685381 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:10.686044 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:10.686052 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:10.686054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:10.686175 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:10.686523 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:10.686529 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:10.686532 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:10.686865 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:10.687191 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:10.687447 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 11:34:10.687708 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":341,"threads_started":1}
W20260924 11:34:10.689320 10540 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49384
W20260924 11:34:10.690471 6992 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 10417: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:34:10.690632 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 10417
I20260924 11:34:10.706141 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 10057
I20260924 11:34:10.707403 10119 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:34:10.809082 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 10057
2026-09-24T11:34:10Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3632 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:34:10 dist-test-slave-cq7c krb5kdc[10562](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:34:10 dist-test-slave-cq7c krb5kdc[10562](info): set up 2 sockets
Sep 24 11:34:10 dist-test-slave-cq7c krb5kdc[10562](info): commencing operation
krb5kdc: starting...
W20260924 11:34:11.907876 6992 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.005s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Sep 24 11:34:12 dist-test-slave-cq7c krb5kdc[10562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:34:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:34:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.212.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
I20260924 11:34:12.107867 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:45589
--webserver_interface=127.6.212.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:36659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:45589
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:34:12.448457 10575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:12.448760 10575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:12.448894 10575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:12.456343 10575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:34:12.456493 10575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:12.456575 10575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:12.456673 10575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:34:12.456748 10575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:34:12.466774 10575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:36659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:45589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:45589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.212.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:34:12.468223 10575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:12.469521 10575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:12.481047 10580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:12.481070 10581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:12.480989 10583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:12.481920 10575 server_base.cc:1034] running on GCE node
I20260924 11:34:12.482774 10575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:12.484314 10575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:12.485546 10575 hybrid_clock.cc:648] HybridClock initialized: now 1790249652485504 us; error 54 us; skew 500 ppm
I20260924 11:34:12.485904 10575 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:34:12 dist-test-slave-cq7c krb5kdc[10562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249652, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:12.488968 10575 init.cc:377] Logged in from keytab as kudu/127.6.212.62@KRBTEST.COM (short username kudu)
I20260924 11:34:12.490424 10575 webserver.cc:460] Webserver started at http://127.6.212.62:41455/ using document root <none> and password file <none>
I20260924 11:34:12.491103 10575 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:12.491248 10575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:12.491591 10575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:34:12.494599 10575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "4ddb2f873d3d4bceb281e20448ddc667"
format_stamp: "Formatted at 2026-09-24 11:34:12 on dist-test-slave-cq7c"
server_key: "9853955d1122a78e64a078e338392bd5"
server_key_iv: "cf26200324a27b66c1f6b9202fe1fe4c"
server_key_version: "encryptionkey@0"
I20260924 11:34:12.495424 10575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "4ddb2f873d3d4bceb281e20448ddc667"
format_stamp: "Formatted at 2026-09-24 11:34:12 on dist-test-slave-cq7c"
server_key: "9853955d1122a78e64a078e338392bd5"
server_key_iv: "cf26200324a27b66c1f6b9202fe1fe4c"
server_key_version: "encryptionkey@0"
I20260924 11:34:12.500653 10575 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 11:34:12.504292 10590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:12.505653 10575 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:34:12.505890 10575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "4ddb2f873d3d4bceb281e20448ddc667"
format_stamp: "Formatted at 2026-09-24 11:34:12 on dist-test-slave-cq7c"
server_key: "9853955d1122a78e64a078e338392bd5"
server_key_iv: "cf26200324a27b66c1f6b9202fe1fe4c"
server_key_version: "encryptionkey@0"
I20260924 11:34:12.506152 10575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:12.527532 10575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:12.531401 10575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:34:12.531675 10575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:12.549803 10575 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.62:45589
I20260924 11:34:12.549850 10641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.62:45589 every 8 connection(s)
I20260924 11:34:12.551131 10575 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:34:12.554742 10642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:34:12.554948 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 10575
I20260924 11:34:12.555146 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:34:12.555820 6992 external_mini_cluster.cc:1428] Setting key b279bf773b088da44e8a52c9121301ff
I20260924 11:34:12.568181 10642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ddb2f873d3d4bceb281e20448ddc667: Bootstrap starting.
Sep 24 11:34:12 dist-test-slave-cq7c krb5kdc[10562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:34:12.571753 10642 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ddb2f873d3d4bceb281e20448ddc667: Neither blocks nor log segments found. Creating new log.
I20260924 11:34:12.572788 10642 log.cc:824] T 00000000000000000000000000000000 P 4ddb2f873d3d4bceb281e20448ddc667: Log is configured to *not* fsync() on all Append() calls
I20260924 11:34:12.575477 10642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ddb2f873d3d4bceb281e20448ddc667: No bootstrap required, opened a new log
I20260924 11:34:12.578960 10645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:12.557668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36932 (local address 127.6.212.62:45589)
0924 11:34:12.558437 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:12.558448 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:12.558490 (+ 42us) server_negotiation.cc:407] Connection header received
0924 11:34:12.560553 (+ 2063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:12.560577 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:12.560959 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:12.561293 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:12.563065 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:12.564445 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:12.566193 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:12.566839 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:12.566962 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:12.571346 (+ 4384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:12.571354 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:12.571425 (+ 71us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:12.571466 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:12.574379 (+ 2913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:12.575063 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:12.575070 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:12.575076 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:12.575247 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:12.575673 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:12.575678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:12.575681 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:12.576393 (+ 712us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:12.576684 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:12.577122 (+ 438us) server_negotiation.cc:299] Negotiation successful
0924 11:34:12.577572 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":359,"threads_started":1}
I20260924 11:34:12.583951 10642 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ddb2f873d3d4bceb281e20448ddc667 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ddb2f873d3d4bceb281e20448ddc667" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 45589 } }
I20260924 11:34:12.584277 10642 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ddb2f873d3d4bceb281e20448ddc667 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:34:12.584353 10642 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ddb2f873d3d4bceb281e20448ddc667 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ddb2f873d3d4bceb281e20448ddc667, State: Initialized, Role: FOLLOWER
I20260924 11:34:12.584810 10642 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ddb2f873d3d4bceb281e20448ddc667 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ddb2f873d3d4bceb281e20448ddc667" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 45589 } }
I20260924 11:34:12.584998 10642 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ddb2f873d3d4bceb281e20448ddc667 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:34:12.585083 10642 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ddb2f873d3d4bceb281e20448ddc667 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:34:12.585207 10642 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ddb2f873d3d4bceb281e20448ddc667 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:34:12.587209 10642 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ddb2f873d3d4bceb281e20448ddc667 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ddb2f873d3d4bceb281e20448ddc667" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 45589 } }
I20260924 11:34:12.587765 10642 leader_election.cc:302] T 00000000000000000000000000000000 P 4ddb2f873d3d4bceb281e20448ddc667 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ddb2f873d3d4bceb281e20448ddc667; no voters:
I20260924 11:34:12.588395 10642 leader_election.cc:288] T 00000000000000000000000000000000 P 4ddb2f873d3d4bceb281e20448ddc667 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:34:12.588511 10647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ddb2f873d3d4bceb281e20448ddc667 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:34:12.589315 10647 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ddb2f873d3d4bceb281e20448ddc667 [term 1 LEADER]: Becoming Leader. State: Replica: 4ddb2f873d3d4bceb281e20448ddc667, State: Running, Role: LEADER
I20260924 11:34:12.589918 10647 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ddb2f873d3d4bceb281e20448ddc667 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ddb2f873d3d4bceb281e20448ddc667" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 45589 } }
I20260924 11:34:12.590649 10642 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ddb2f873d3d4bceb281e20448ddc667 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:34:12.592828 10649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ddb2f873d3d4bceb281e20448ddc667 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ddb2f873d3d4bceb281e20448ddc667. Latest consensus state: current_term: 1 leader_uuid: "4ddb2f873d3d4bceb281e20448ddc667" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ddb2f873d3d4bceb281e20448ddc667" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 45589 } } }
I20260924 11:34:12.593263 10649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ddb2f873d3d4bceb281e20448ddc667 [sys.catalog]: This master's current role is: LEADER
I20260924 11:34:12.593830 10648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ddb2f873d3d4bceb281e20448ddc667 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ddb2f873d3d4bceb281e20448ddc667" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ddb2f873d3d4bceb281e20448ddc667" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 45589 } } }
I20260924 11:34:12.594175 10648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ddb2f873d3d4bceb281e20448ddc667 [sys.catalog]: This master's current role is: LEADER
I20260924 11:34:12.596189 10652 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:34:12.601179 10652 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:34:12.608774 10652 catalog_manager.cc:1347] Generated new cluster ID: 06953016ba5546fa92eab93da2357a8e
I20260924 11:34:12.608878 10652 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:34:12.628938 10652 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:34:12.630800 10652 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:34:12.649785 10652 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ddb2f873d3d4bceb281e20448ddc667: Generated new TSK 0
I20260924 11:34:12.650698 10652 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.212.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab.
Entry for principal kudu/127.6.212.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab.
I20260924 11:34:12.710764 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.1:0
--local_ip_for_outbound_sockets=127.6.212.1
--webserver_interface=127.6.212.1
--webserver_port=0
--tserver_master_addrs=127.6.212.62:45589
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab
--principal=kudu/127.6.212.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:36659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:34:13.049242 10668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:13.049516 10668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:13.049680 10668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:13.056811 10668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:13.056915 10668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:13.057025 10668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.1
I20260924 11:34:13.066697 10668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:36659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab
--principal=kudu/127.6.212.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.212.1
--webserver_port=0
--tserver_master_addrs=127.6.212.62:45589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.1
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:34:13.068011 10668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:13.069320 10668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:13.082849 10676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:13.082890 10673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:13.082991 10674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:13.083803 10668 server_base.cc:1034] running on GCE node
I20260924 11:34:13.084465 10668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:13.085495 10668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:13.086764 10668 hybrid_clock.cc:648] HybridClock initialized: now 1790249653086702 us; error 40 us; skew 500 ppm
I20260924 11:34:13.087206 10668 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:34:13 dist-test-slave-cq7c krb5kdc[10562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249653, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:13.090624 10668 init.cc:377] Logged in from keytab as kudu/127.6.212.1@KRBTEST.COM (short username kudu)
I20260924 11:34:13.092418 10668 webserver.cc:460] Webserver started at http://127.6.212.1:42687/ using document root <none> and password file <none>
I20260924 11:34:13.093169 10668 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:13.093321 10668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:13.093667 10668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:34:13.097455 10668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-0/data/instance:
uuid: "c15f8cf4f2f04fe68f55b25960aba24f"
format_stamp: "Formatted at 2026-09-24 11:34:13 on dist-test-slave-cq7c"
server_key: "815d129f02852163aa81db76ad04b6f2"
server_key_iv: "40d36c59c3504a5063b39c8b70ebb165"
server_key_version: "encryptionkey@0"
I20260924 11:34:13.098426 10668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance:
uuid: "c15f8cf4f2f04fe68f55b25960aba24f"
format_stamp: "Formatted at 2026-09-24 11:34:13 on dist-test-slave-cq7c"
server_key: "815d129f02852163aa81db76ad04b6f2"
server_key_iv: "40d36c59c3504a5063b39c8b70ebb165"
server_key_version: "encryptionkey@0"
I20260924 11:34:13.104434 10668 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 11:34:13.108273 10683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:13.109843 10668 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:34:13.110021 10668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-0/wal
uuid: "c15f8cf4f2f04fe68f55b25960aba24f"
format_stamp: "Formatted at 2026-09-24 11:34:13 on dist-test-slave-cq7c"
server_key: "815d129f02852163aa81db76ad04b6f2"
server_key_iv: "40d36c59c3504a5063b39c8b70ebb165"
server_key_version: "encryptionkey@0"
I20260924 11:34:13.110213 10668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:13.136358 10668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:13.140579 10668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:34:13.140888 10668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:13.141942 10668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:34:13.143335 10668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:34:13.143471 10668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:13.143631 10668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:34:13.143697 10668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:13.177652 10668 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.1:43239
I20260924 11:34:13.177692 10795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.1:43239 every 8 connection(s)
I20260924 11:34:13.179021 10668 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
I20260924 11:34:13.182113 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 10668
I20260924 11:34:13.182317 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance
I20260924 11:34:13.183236 6992 external_mini_cluster.cc:1428] Setting key ab7738b528af0b4980abf15c872e9cd8
Sep 24 11:34:13 dist-test-slave-cq7c krb5kdc[10562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249653, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:34:13.201340 10798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:13.181923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:47789 (local address 127.6.212.62:45589)
0924 11:34:13.182379 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:13.182383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:13.183698 (+ 1315us) server_negotiation.cc:407] Connection header received
0924 11:34:13.185738 (+ 2040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:13.185743 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:13.185849 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:13.185967 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:13.187457 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:13.188170 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:13.189693 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:13.190172 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:13.190314 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:13.194340 (+ 4026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:13.194349 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:13.194357 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:13.194402 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:13.198017 (+ 3615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:13.198866 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:13.198872 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:13.198876 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:13.199023 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:13.199447 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:13.199454 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:13.199461 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:13.199855 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:13.200030 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:13.200676 (+ 646us) server_negotiation.cc:299] Negotiation successful
0924 11:34:13.200909 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":260,"threads_started":1}
I20260924 11:34:13.202100 10799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:13.182677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:45589 (local address 127.6.212.1:47789)
0924 11:34:13.183545 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:13.183588 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:13.185483 (+ 1895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:13.186198 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:13.186213 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:13.186698 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:13.187275 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:13.187292 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:13.188325 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:13.188335 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:13.189544 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:13.189555 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:13.190407 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:13.190418 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:13.190694 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:13.191476 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:13.191547 (+ 71us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:13.194113 (+ 2566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:13.198155 (+ 4042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:13.198162 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:13.198175 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:13.198730 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:13.199130 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:13.199133 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:13.199135 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:13.199303 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:13.199986 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:13.199992 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:13.200305 (+ 313us) client_negotiation.cc:769] Sending connection context
0924 11:34:13.200522 (+ 217us) client_negotiation.cc:240] Negotiation successful
0924 11:34:13.200904 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":339,"threads_started":1}
WARNING: no policy specified for kudu/127.6.212.2@KRBTEST.COM; defaulting to no policy
I20260924 11:34:13.203619 10796 heartbeater.cc:347] Connected to a master server at 127.6.212.62:45589
I20260924 11:34:13.203922 10796 heartbeater.cc:464] Registering TS with master...
I20260924 11:34:13.204555 10796 heartbeater.cc:511] Master 127.6.212.62:45589 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.2@KRBTEST.COM" created.
I20260924 11:34:13.206817 10607 ts_manager.cc:194] Registered new tserver with Master: c15f8cf4f2f04fe68f55b25960aba24f (127.6.212.1:43239)
I20260924 11:34:13.208948 10607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.1@KRBTEST.COM'} at 127.6.212.1:47789
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab.
Entry for principal kudu/127.6.212.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab.
I20260924 11:34:13.228891 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.2:0
--local_ip_for_outbound_sockets=127.6.212.2
--webserver_interface=127.6.212.2
--webserver_port=0
--tserver_master_addrs=127.6.212.62:45589
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab
--principal=kudu/127.6.212.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:36659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:34:13.545861 10802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:13.546180 10802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:13.546350 10802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:13.553584 10802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:13.553730 10802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:13.553918 10802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.2
I20260924 11:34:13.563623 10802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:36659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab
--principal=kudu/127.6.212.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.212.2
--webserver_port=0
--tserver_master_addrs=127.6.212.62:45589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.2
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:34:13.564867 10802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:13.566203 10802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:13.579558 10807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:13.579617 10810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:13.579644 10808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:13.580492 10802 server_base.cc:1034] running on GCE node
I20260924 11:34:13.581100 10802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:13.582098 10802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:13.583329 10802 hybrid_clock.cc:648] HybridClock initialized: now 1790249653583282 us; error 54 us; skew 500 ppm
I20260924 11:34:13.583707 10802 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:34:13 dist-test-slave-cq7c krb5kdc[10562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249653, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:13.587060 10802 init.cc:377] Logged in from keytab as kudu/127.6.212.2@KRBTEST.COM (short username kudu)
I20260924 11:34:13.588546 10802 webserver.cc:460] Webserver started at http://127.6.212.2:41005/ using document root <none> and password file <none>
I20260924 11:34:13.589300 10802 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:13.589449 10802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:13.589775 10802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:34:13.592988 10802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-1/data/instance:
uuid: "b447a90a96ec4291afd8fab017a2ec66"
format_stamp: "Formatted at 2026-09-24 11:34:13 on dist-test-slave-cq7c"
server_key: "a82ac3f5bbbe99a44fea807d3d50ae01"
server_key_iv: "32990140e9962cec4d5811f7aa496429"
server_key_version: "encryptionkey@0"
I20260924 11:34:13.593808 10802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance:
uuid: "b447a90a96ec4291afd8fab017a2ec66"
format_stamp: "Formatted at 2026-09-24 11:34:13 on dist-test-slave-cq7c"
server_key: "a82ac3f5bbbe99a44fea807d3d50ae01"
server_key_iv: "32990140e9962cec4d5811f7aa496429"
server_key_version: "encryptionkey@0"
I20260924 11:34:13.599310 10802 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 11:34:13.602927 10817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:13.604396 10802 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:34:13.604573 10802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-1/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-1/wal
uuid: "b447a90a96ec4291afd8fab017a2ec66"
format_stamp: "Formatted at 2026-09-24 11:34:13 on dist-test-slave-cq7c"
server_key: "a82ac3f5bbbe99a44fea807d3d50ae01"
server_key_iv: "32990140e9962cec4d5811f7aa496429"
server_key_version: "encryptionkey@0"
I20260924 11:34:13.604795 10802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:13.631162 10802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:13.635905 10802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:34:13.636219 10802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:13.637284 10802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:34:13.638656 10802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:34:13.638759 10802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:13.638909 10802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:34:13.638984 10802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:13.672600 10802 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.2:39835
I20260924 11:34:13.672631 10929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.2:39835 every 8 connection(s)
I20260924 11:34:13.673986 10802 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
I20260924 11:34:13.677543 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 10802
I20260924 11:34:13.677708 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance
I20260924 11:34:13.678117 6992 external_mini_cluster.cc:1428] Setting key 8200e9df9194b38e65c0aa57177a842b
Sep 24 11:34:13 dist-test-slave-cq7c krb5kdc[10562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249653, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.2@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:34:13.695904 10798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:13.676581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:41259 (local address 127.6.212.62:45589)
0924 11:34:13.676765 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:13.676770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:13.679330 (+ 2560us) server_negotiation.cc:407] Connection header received
0924 11:34:13.681207 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:13.681212 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:13.681293 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:13.681399 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:13.682830 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:13.683559 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:13.685163 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:13.685616 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:13.685724 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:13.689849 (+ 4125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:13.689854 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:13.689858 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:13.689888 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:13.692809 (+ 2921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:13.693562 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:13.693573 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:13.693578 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:13.693698 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:13.694130 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:13.694135 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:13.694139 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:13.694413 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:13.694572 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:13.695284 (+ 712us) server_negotiation.cc:299] Negotiation successful
0924 11:34:13.695538 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
WARNING: no policy specified for kudu/127.6.212.3@KRBTEST.COM; defaulting to no policy
I20260924 11:34:13.697007 10932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:13.677119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:45589 (local address 127.6.212.2:41259)
0924 11:34:13.679166 (+ 2047us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:13.679219 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:13.680958 (+ 1739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:13.681622 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:13.681631 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:13.682124 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:13.682675 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:13.682692 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:13.683702 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:13.683707 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:13.684938 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:13.684956 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:13.685849 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:13.685858 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:13.686178 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:13.686926 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:13.686960 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:13.689607 (+ 2647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:13.692949 (+ 3342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:13.692959 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:13.692976 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:13.693432 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:13.693809 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:13.693815 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:13.693820 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:13.694006 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:13.694548 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:13.694557 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:13.694907 (+ 350us) client_negotiation.cc:769] Sending connection context
0924 11:34:13.695189 (+ 282us) client_negotiation.cc:240] Negotiation successful
0924 11:34:13.695780 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1717,"thread_start_us":1526,"threads_started":1}
I20260924 11:34:13.698707 10930 heartbeater.cc:347] Connected to a master server at 127.6.212.62:45589
I20260924 11:34:13.699034 10930 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.3@KRBTEST.COM" created.
I20260924 11:34:13.699836 10930 heartbeater.cc:511] Master 127.6.212.62:45589 requested a full tablet report, sending...
I20260924 11:34:13.701319 10607 ts_manager.cc:194] Registered new tserver with Master: b447a90a96ec4291afd8fab017a2ec66 (127.6.212.2:39835)
I20260924 11:34:13.702124 10607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.2@KRBTEST.COM'} at 127.6.212.2:41259
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab.
Entry for principal kudu/127.6.212.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab.
I20260924 11:34:13.727046 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.3:0
--local_ip_for_outbound_sockets=127.6.212.3
--webserver_interface=127.6.212.3
--webserver_port=0
--tserver_master_addrs=127.6.212.62:45589
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab
--principal=kudu/127.6.212.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:36659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:34:14.055294 10935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:14.055675 10935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:14.055847 10935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:14.062968 10935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:14.063103 10935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:14.063290 10935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.3
I20260924 11:34:14.072980 10935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:36659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab
--principal=kudu/127.6.212.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.212.3
--webserver_port=0
--tserver_master_addrs=127.6.212.62:45589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.3
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:34:14.074355 10935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:14.075690 10935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:14.089097 10941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:14.089181 10940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:14.089201 10943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:14.089866 10935 server_base.cc:1034] running on GCE node
I20260924 11:34:14.090523 10935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:14.091584 10935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:14.092801 10935 hybrid_clock.cc:648] HybridClock initialized: now 1790249654092740 us; error 49 us; skew 500 ppm
I20260924 11:34:14.093122 10935 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:34:14 dist-test-slave-cq7c krb5kdc[10562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249654, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:14.096365 10935 init.cc:377] Logged in from keytab as kudu/127.6.212.3@KRBTEST.COM (short username kudu)
I20260924 11:34:14.098053 10935 webserver.cc:460] Webserver started at http://127.6.212.3:40575/ using document root <none> and password file <none>
I20260924 11:34:14.099000 10935 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:14.099133 10935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:14.099542 10935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:34:14.103974 10935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-2/data/instance:
uuid: "b5ffe22e337f4cf88c2e7b5249485e96"
format_stamp: "Formatted at 2026-09-24 11:34:14 on dist-test-slave-cq7c"
server_key: "5f7fae7b1a617bfcb011897f43081489"
server_key_iv: "97fda4c366d7d30420b6fdb13c1ecb89"
server_key_version: "encryptionkey@0"
I20260924 11:34:14.105072 10935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance:
uuid: "b5ffe22e337f4cf88c2e7b5249485e96"
format_stamp: "Formatted at 2026-09-24 11:34:14 on dist-test-slave-cq7c"
server_key: "5f7fae7b1a617bfcb011897f43081489"
server_key_iv: "97fda4c366d7d30420b6fdb13c1ecb89"
server_key_version: "encryptionkey@0"
I20260924 11:34:14.111147 10935 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260924 11:34:14.114709 10950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:14.116509 10935 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:34:14.116689 10935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-2/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-2/wal
uuid: "b5ffe22e337f4cf88c2e7b5249485e96"
format_stamp: "Formatted at 2026-09-24 11:34:14 on dist-test-slave-cq7c"
server_key: "5f7fae7b1a617bfcb011897f43081489"
server_key_iv: "97fda4c366d7d30420b6fdb13c1ecb89"
server_key_version: "encryptionkey@0"
I20260924 11:34:14.116899 10935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:14.138201 10935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:14.142473 10935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:34:14.142765 10935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:14.143774 10935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:34:14.145090 10935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:34:14.145192 10935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:14.145331 10935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:34:14.145399 10935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:14.177668 10935 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.3:46347
I20260924 11:34:14.177702 11062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.3:46347 every 8 connection(s)
I20260924 11:34:14.178944 10935 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
I20260924 11:34:14.186647 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 10935
I20260924 11:34:14.186836 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance
I20260924 11:34:14.187260 6992 external_mini_cluster.cc:1428] Setting key 75558451304b51d69a3ba35569223ea3
Sep 24 11:34:14 dist-test-slave-cq7c krb5kdc[10562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249654, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.3@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:34:14.204545 10798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:14.181699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:35775 (local address 127.6.212.62:45589)
0924 11:34:14.181889 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:14.181893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:14.183207 (+ 1314us) server_negotiation.cc:407] Connection header received
0924 11:34:14.185079 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:14.185085 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:14.185182 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:14.185288 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:14.186738 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:14.187627 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:14.189582 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:14.190052 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:14.190182 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:14.198890 (+ 8708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:14.198896 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:14.198899 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:14.198930 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:14.201754 (+ 2824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:14.202642 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:14.202646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:14.202648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:14.202770 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:14.203120 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:14.203125 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:14.203127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:14.203414 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:14.203520 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:14.204160 (+ 640us) server_negotiation.cc:299] Negotiation successful
0924 11:34:14.204306 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 11:34:14.205538 11065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:14.182254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:45589 (local address 127.6.212.3:35775)
0924 11:34:14.182990 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:14.183031 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:14.184782 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:14.185518 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:14.185526 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:14.186023 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:14.186571 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:14.186591 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:14.187763 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:14.187768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:14.189441 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:14.189452 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:14.190279 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:14.190309 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:14.190581 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:14.196420 (+ 5839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:14.196444 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:14.198669 (+ 2225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:14.201911 (+ 3242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:14.201920 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:14.201941 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:14.202525 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:14.202860 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:14.202864 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:14.202866 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:14.203032 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:14.203512 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:14.203517 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:14.203850 (+ 333us) client_negotiation.cc:769] Sending connection context
0924 11:34:14.204078 (+ 228us) client_negotiation.cc:240] Negotiation successful
0924 11:34:14.204461 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":282,"threads_started":1}
I20260924 11:34:14.206799 11063 heartbeater.cc:347] Connected to a master server at 127.6.212.62:45589
I20260924 11:34:14.207063 11063 heartbeater.cc:464] Registering TS with master...
I20260924 11:34:14.207789 11063 heartbeater.cc:511] Master 127.6.212.62:45589 requested a full tablet report, sending...
I20260924 11:34:14.209096 10607 ts_manager.cc:194] Registered new tserver with Master: b5ffe22e337f4cf88c2e7b5249485e96 (127.6.212.3:46347)
I20260924 11:34:14.209971 10607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.3@KRBTEST.COM'} at 127.6.212.3:35775
I20260924 11:34:14.212355 10796 heartbeater.cc:503] Master 127.6.212.62:45589 was elected leader, sending a full tablet report...
I20260924 11:34:14.214550 6992 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 11:34:14 dist-test-slave-cq7c krb5kdc[10562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 11:34:14 dist-test-slave-cq7c krb5kdc[10562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.1@KRBTEST.COM
I20260924 11:34:14.262481 11070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:14.248373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52768 (local address 127.6.212.1:43239)
0924 11:34:14.249026 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:14.249033 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:14.249057 (+ 24us) server_negotiation.cc:407] Connection header received
0924 11:34:14.249113 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:14.249119 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:14.249330 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:14.249510 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:14.250838 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:14.251895 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:14.253137 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:14.253689 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:14.253835 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:14.257646 (+ 3811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:14.257656 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:14.257671 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:14.257705 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:14.260272 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:14.260837 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:14.260844 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:14.260846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:14.260960 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:14.261257 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:14.261260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:14.261262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:14.261565 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:14.261784 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:14.262014 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 11:34:14.262205 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":290,"threads_started":1}
Sep 24 11:34:14 dist-test-slave-cq7c krb5kdc[10562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:34:14.279279 10798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:14.266449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36948 (local address 127.6.212.62:45589)
0924 11:34:14.266606 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:14.266609 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:14.266896 (+ 287us) server_negotiation.cc:407] Connection header received
0924 11:34:14.267016 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:14.267020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:14.267080 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:14.267160 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:14.268412 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:14.269121 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:14.270353 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:14.270745 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:14.270830 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:14.274647 (+ 3817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:14.274651 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:14.274654 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:14.274685 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:14.277134 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:14.277666 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:14.277670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:14.277672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:14.277796 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:14.278071 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:14.278074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:14.278076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:14.278354 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:14.278525 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:14.278840 (+ 315us) server_negotiation.cc:299] Negotiation successful
0924 11:34:14.279036 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260924 11:34:14.279872 10607 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36948
I20260924 11:34:14.280890 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 10668
I20260924 11:34:14.293058 11070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:14.281731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52770 (local address 127.6.212.1:43239)
0924 11:34:14.281943 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:14.281949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:14.282096 (+ 147us) server_negotiation.cc:407] Connection header received
0924 11:34:14.282251 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:14.282255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:14.282339 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:14.282414 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:14.283573 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:14.284308 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:14.285413 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:14.285792 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:14.285883 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:14.288118 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:14.288123 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:14.288126 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:14.288157 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:14.290795 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:14.291458 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:14.291464 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:14.291468 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:14.291585 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:14.291929 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:14.291933 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:14.291935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:14.292198 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:14.292395 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:14.292587 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 11:34:14.292748 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 11:34:14.293720 10791 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:34:14.376606 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 10668
I20260924 11:34:14.392835 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 10802
Sep 24 11:34:14 dist-test-slave-cq7c krb5kdc[10562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.2@KRBTEST.COM
I20260924 11:34:14.408428 11075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:14.393775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43880 (local address 127.6.212.2:39835)
0924 11:34:14.394311 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:14.394319 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:14.394343 (+ 24us) server_negotiation.cc:407] Connection header received
0924 11:34:14.394400 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:14.394407 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:14.394612 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:14.394740 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:14.395974 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:14.397007 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:14.398284 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:14.398877 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:14.398975 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:14.402893 (+ 3918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:14.402902 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:14.402916 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:14.402953 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:14.405648 (+ 2695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:14.406275 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:14.406281 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:14.406283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:14.406397 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:14.406699 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:14.406702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:14.406704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:14.407002 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:14.407242 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:14.407790 (+ 548us) server_negotiation.cc:299] Negotiation successful
0924 11:34:14.407984 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":256,"threads_started":1}
I20260924 11:34:14.409871 10925 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:34:14.492175 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 10802
I20260924 11:34:14.508345 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 10935
Sep 24 11:34:14 dist-test-slave-cq7c krb5kdc[10562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.3@KRBTEST.COM
I20260924 11:34:14.524219 11077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:14.509272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44566 (local address 127.6.212.3:46347)
0924 11:34:14.509873 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:14.509880 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:14.509909 (+ 29us) server_negotiation.cc:407] Connection header received
0924 11:34:14.509986 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:14.509991 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:14.510218 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:14.510344 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:14.511705 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:14.512713 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:14.513890 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:14.514423 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:14.514514 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:14.518612 (+ 4098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:14.518622 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:14.518637 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:14.518671 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:14.521567 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:14.522205 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:14.522212 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:14.522214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:14.522330 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:14.522679 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:14.522683 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:14.522685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:14.522997 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:14.523284 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:14.523624 (+ 340us) server_negotiation.cc:299] Negotiation successful
0924 11:34:14.523870 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":280,"threads_started":1}
I20260924 11:34:14.525475 11058 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:34:14.606916 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 10935
I20260924 11:34:14.622402 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 10575
I20260924 11:34:14.623605 10637 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:34:14.725134 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 10575
2026-09-24T11:34:14Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3915 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:34:14 dist-test-slave-cq7c krb5kdc[11082](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:34:14 dist-test-slave-cq7c krb5kdc[11082](info): set up 2 sockets
Sep 24 11:34:14 dist-test-slave-cq7c krb5kdc[11082](info): commencing operation
krb5kdc: starting...
W20260924 11:34:15.819931 6992 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Sep 24 11:34:15 dist-test-slave-cq7c krb5kdc[11082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:34:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:34:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.212.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
I20260924 11:34:16.018872 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:38925
--webserver_interface=127.6.212.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:44677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:38925
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:34:16.338807 11095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:16.339148 11095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:16.339300 11095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:16.346513 11095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:34:16.346654 11095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:16.346764 11095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:16.346860 11095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:34:16.346943 11095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:34:16.356765 11095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:44677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:38925
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:38925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.212.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:34:16.358083 11095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:16.359421 11095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:16.371032 11101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:16.371114 11103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:16.371133 11100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:16.371788 11095 server_base.cc:1034] running on GCE node
I20260924 11:34:16.372537 11095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:16.373917 11095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:16.375190 11095 hybrid_clock.cc:648] HybridClock initialized: now 1790249656375130 us; error 54 us; skew 500 ppm
I20260924 11:34:16.375587 11095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:34:16 dist-test-slave-cq7c krb5kdc[11082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249656, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:16.378556 11095 init.cc:377] Logged in from keytab as kudu/127.6.212.62@KRBTEST.COM (short username kudu)
I20260924 11:34:16.379715 11095 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:16.379837 11095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:16.380179 11095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:34:16.383133 11095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "6d81c08e1dfa43b19f2532ad89b969e9"
format_stamp: "Formatted at 2026-09-24 11:34:16 on dist-test-slave-cq7c"
server_key: "dc6a5c99b3f09977ea206f58f45d8dd3"
server_key_iv: "b5d009eade0202491c5b751016528a7b"
server_key_version: "encryptionkey@0"
I20260924 11:34:16.383946 11095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "6d81c08e1dfa43b19f2532ad89b969e9"
format_stamp: "Formatted at 2026-09-24 11:34:16 on dist-test-slave-cq7c"
server_key: "dc6a5c99b3f09977ea206f58f45d8dd3"
server_key_iv: "b5d009eade0202491c5b751016528a7b"
server_key_version: "encryptionkey@0"
I20260924 11:34:16.388955 11095 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 11:34:16.392630 11109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:16.393864 11095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:34:16.394038 11095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "6d81c08e1dfa43b19f2532ad89b969e9"
format_stamp: "Formatted at 2026-09-24 11:34:16 on dist-test-slave-cq7c"
server_key: "dc6a5c99b3f09977ea206f58f45d8dd3"
server_key_iv: "b5d009eade0202491c5b751016528a7b"
server_key_version: "encryptionkey@0"
I20260924 11:34:16.394253 11095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:16.409920 11095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:16.413738 11095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:34:16.413992 11095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:16.431452 11095 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.62:38925
I20260924 11:34:16.431471 11160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.62:38925 every 8 connection(s)
I20260924 11:34:16.432318 11095 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:34:16.435942 11161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:34:16.436928 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 11095
I20260924 11:34:16.437155 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:34:16.437783 6992 external_mini_cluster.cc:1428] Setting key f64076b399dab35dc00a4572de77a7f9
I20260924 11:34:16.449330 11161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d81c08e1dfa43b19f2532ad89b969e9: Bootstrap starting.
Sep 24 11:34:16 dist-test-slave-cq7c krb5kdc[11082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:34:16.453117 11161 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d81c08e1dfa43b19f2532ad89b969e9: Neither blocks nor log segments found. Creating new log.
I20260924 11:34:16.454293 11161 log.cc:824] T 00000000000000000000000000000000 P 6d81c08e1dfa43b19f2532ad89b969e9: Log is configured to *not* fsync() on all Append() calls
I20260924 11:34:16.457435 11161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d81c08e1dfa43b19f2532ad89b969e9: No bootstrap required, opened a new log
I20260924 11:34:16.460577 11164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:16.439732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58698 (local address 127.6.212.62:38925)
0924 11:34:16.440524 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:16.440536 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:16.440581 (+ 45us) server_negotiation.cc:407] Connection header received
0924 11:34:16.442143 (+ 1562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:16.442167 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:16.442549 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:16.442870 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:16.444644 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:16.445970 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:16.447860 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:16.448542 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:16.448683 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:16.453407 (+ 4724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:16.453415 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:16.453428 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:16.453469 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:16.456687 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:16.457345 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:16.457351 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:16.457357 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:16.457515 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:16.457937 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:16.457940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:16.457942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:16.458480 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:16.458699 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:16.459172 (+ 473us) server_negotiation.cc:299] Negotiation successful
0924 11:34:16.459602 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":591,"thread_start_us":393,"threads_started":1}
I20260924 11:34:16.465785 11161 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d81c08e1dfa43b19f2532ad89b969e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d81c08e1dfa43b19f2532ad89b969e9" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 38925 } }
I20260924 11:34:16.466126 11161 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d81c08e1dfa43b19f2532ad89b969e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:34:16.466223 11161 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d81c08e1dfa43b19f2532ad89b969e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d81c08e1dfa43b19f2532ad89b969e9, State: Initialized, Role: FOLLOWER
I20260924 11:34:16.466790 11161 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d81c08e1dfa43b19f2532ad89b969e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d81c08e1dfa43b19f2532ad89b969e9" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 38925 } }
I20260924 11:34:16.467021 11161 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d81c08e1dfa43b19f2532ad89b969e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:34:16.467147 11161 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d81c08e1dfa43b19f2532ad89b969e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:34:16.467329 11161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d81c08e1dfa43b19f2532ad89b969e9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:34:16.469524 11161 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d81c08e1dfa43b19f2532ad89b969e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d81c08e1dfa43b19f2532ad89b969e9" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 38925 } }
I20260924 11:34:16.469954 11161 leader_election.cc:302] T 00000000000000000000000000000000 P 6d81c08e1dfa43b19f2532ad89b969e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d81c08e1dfa43b19f2532ad89b969e9; no voters:
I20260924 11:34:16.470448 11161 leader_election.cc:288] T 00000000000000000000000000000000 P 6d81c08e1dfa43b19f2532ad89b969e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:34:16.470615 11166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d81c08e1dfa43b19f2532ad89b969e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:34:16.471330 11166 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d81c08e1dfa43b19f2532ad89b969e9 [term 1 LEADER]: Becoming Leader. State: Replica: 6d81c08e1dfa43b19f2532ad89b969e9, State: Running, Role: LEADER
I20260924 11:34:16.471866 11166 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d81c08e1dfa43b19f2532ad89b969e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d81c08e1dfa43b19f2532ad89b969e9" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 38925 } }
I20260924 11:34:16.472553 11161 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d81c08e1dfa43b19f2532ad89b969e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:34:16.474290 11168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d81c08e1dfa43b19f2532ad89b969e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d81c08e1dfa43b19f2532ad89b969e9. Latest consensus state: current_term: 1 leader_uuid: "6d81c08e1dfa43b19f2532ad89b969e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d81c08e1dfa43b19f2532ad89b969e9" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 38925 } } }
I20260924 11:34:16.474648 11168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d81c08e1dfa43b19f2532ad89b969e9 [sys.catalog]: This master's current role is: LEADER
I20260924 11:34:16.474661 11167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d81c08e1dfa43b19f2532ad89b969e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d81c08e1dfa43b19f2532ad89b969e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d81c08e1dfa43b19f2532ad89b969e9" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 38925 } } }
I20260924 11:34:16.475643 11167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d81c08e1dfa43b19f2532ad89b969e9 [sys.catalog]: This master's current role is: LEADER
I20260924 11:34:16.484331 11171 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:34:16.489225 11171 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:34:16.497684 11171 catalog_manager.cc:1347] Generated new cluster ID: 5edf6e57cf614a708f0bef289eeea0d5
I20260924 11:34:16.497795 11171 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:34:16.520351 11171 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:34:16.521654 11171 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:34:16.527499 11171 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d81c08e1dfa43b19f2532ad89b969e9: Generated new TSK 0
I20260924 11:34:16.528316 11171 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.212.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab.
Entry for principal kudu/127.6.212.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab.
I20260924 11:34:16.579107 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.1:0
--local_ip_for_outbound_sockets=127.6.212.1
--webserver_interface=127.6.212.1
--webserver_port=0
--tserver_master_addrs=127.6.212.62:38925
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab
--principal=kudu/127.6.212.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:44677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:34:16.910634 11188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:16.910938 11188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:16.911033 11188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:16.918588 11188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:16.918701 11188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:16.918854 11188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.1
I20260924 11:34:16.928747 11188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:44677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab
--principal=kudu/127.6.212.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.212.1
--webserver_port=0
--tserver_master_addrs=127.6.212.62:38925
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.1
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:34:16.930138 11188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:16.931511 11188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:16.945714 11193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:16.945783 11196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:16.945801 11194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:16.946798 11188 server_base.cc:1034] running on GCE node
I20260924 11:34:16.947515 11188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:16.948480 11188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:16.949667 11188 hybrid_clock.cc:648] HybridClock initialized: now 1790249656949605 us; error 52 us; skew 500 ppm
I20260924 11:34:16.949968 11188 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:34:16 dist-test-slave-cq7c krb5kdc[11082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249656, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:16.953022 11188 init.cc:377] Logged in from keytab as kudu/127.6.212.1@KRBTEST.COM (short username kudu)
I20260924 11:34:16.954100 11188 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:16.954228 11188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:16.954523 11188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:34:16.957396 11188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-0/data/instance:
uuid: "23b0c4cf9b8045b99dda67b8839ed2d3"
format_stamp: "Formatted at 2026-09-24 11:34:16 on dist-test-slave-cq7c"
server_key: "c855ea67ae79e4d2ed5bcf21dc4ce90c"
server_key_iv: "02b97c156356caaed016d10d723b31a6"
server_key_version: "encryptionkey@0"
I20260924 11:34:16.958159 11188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance:
uuid: "23b0c4cf9b8045b99dda67b8839ed2d3"
format_stamp: "Formatted at 2026-09-24 11:34:16 on dist-test-slave-cq7c"
server_key: "c855ea67ae79e4d2ed5bcf21dc4ce90c"
server_key_iv: "02b97c156356caaed016d10d723b31a6"
server_key_version: "encryptionkey@0"
I20260924 11:34:16.963661 11188 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 11:34:16.967988 11202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:16.969439 11188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:34:16.969615 11188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-0/wal
uuid: "23b0c4cf9b8045b99dda67b8839ed2d3"
format_stamp: "Formatted at 2026-09-24 11:34:16 on dist-test-slave-cq7c"
server_key: "c855ea67ae79e4d2ed5bcf21dc4ce90c"
server_key_iv: "02b97c156356caaed016d10d723b31a6"
server_key_version: "encryptionkey@0"
I20260924 11:34:16.969831 11188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:16.995872 11188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:17.000057 11188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:34:17.000331 11188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:17.001252 11188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:34:17.002547 11188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:34:17.002667 11188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:17.002782 11188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:34:17.002863 11188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:17.035109 11188 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.1:35479
I20260924 11:34:17.035163 11314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.1:35479 every 8 connection(s)
I20260924 11:34:17.035992 11188 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
I20260924 11:34:17.038054 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 11188
I20260924 11:34:17.038262 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance
I20260924 11:34:17.039119 6992 external_mini_cluster.cc:1428] Setting key e27fc04d8453cef8c771e50bf666c326
Sep 24 11:34:17 dist-test-slave-cq7c krb5kdc[11082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249656, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.212.2@KRBTEST.COM; defaulting to no policy
I20260924 11:34:17.057292 11317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:17.038711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:50187 (local address 127.6.212.62:38925)
0924 11:34:17.039176 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:17.039183 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:17.040253 (+ 1070us) server_negotiation.cc:407] Connection header received
0924 11:34:17.042055 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:17.042064 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:17.042154 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:17.042282 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:17.044075 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:17.045240 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:17.046906 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:17.047479 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:17.047629 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:17.051309 (+ 3680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:17.051315 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:17.051318 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:17.051351 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:17.054110 (+ 2759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:17.054873 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:17.054881 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:17.054885 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:17.055027 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:17.055499 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:17.055506 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:17.055512 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:17.055778 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:17.055936 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:17.056640 (+ 704us) server_negotiation.cc:299] Negotiation successful
0924 11:34:17.056858 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":235,"threads_started":1}
I20260924 11:34:17.058212 11318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:17.039250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:38925 (local address 127.6.212.1:50187)
0924 11:34:17.040096 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:17.040148 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:17.041792 (+ 1644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:17.042554 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:17.042563 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:17.043154 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:17.043894 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:17.043914 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:17.045402 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:17.045406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:17.046721 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:17.046733 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:17.047729 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:17.047738 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:17.047977 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:17.048691 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:17.048713 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:17.051084 (+ 2371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:17.054251 (+ 3167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:17.054261 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:17.054276 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:17.054750 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:17.055148 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:17.055154 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:17.055158 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:17.055341 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:17.055919 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:17.055928 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:17.056268 (+ 340us) client_negotiation.cc:769] Sending connection context
0924 11:34:17.056538 (+ 270us) client_negotiation.cc:240] Negotiation successful
0924 11:34:17.057050 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":359,"threads_started":1}
I20260924 11:34:17.059809 11315 heartbeater.cc:347] Connected to a master server at 127.6.212.62:38925
I20260924 11:34:17.060108 11315 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.2@KRBTEST.COM" created.
I20260924 11:34:17.060794 11315 heartbeater.cc:511] Master 127.6.212.62:38925 requested a full tablet report, sending...
I20260924 11:34:17.063092 11126 ts_manager.cc:194] Registered new tserver with Master: 23b0c4cf9b8045b99dda67b8839ed2d3 (127.6.212.1:35479)
I20260924 11:34:17.065243 11126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.1@KRBTEST.COM'} at 127.6.212.1:50187
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab.
Entry for principal kudu/127.6.212.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab.
I20260924 11:34:17.086826 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.2:0
--local_ip_for_outbound_sockets=127.6.212.2
--webserver_interface=127.6.212.2
--webserver_port=0
--tserver_master_addrs=127.6.212.62:38925
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab
--principal=kudu/127.6.212.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:44677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:34:17.409077 11321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:17.409428 11321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:17.409615 11321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:17.417718 11321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:17.417862 11321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:17.418049 11321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.2
I20260924 11:34:17.427984 11321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:44677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab
--principal=kudu/127.6.212.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.212.2
--webserver_port=0
--tserver_master_addrs=127.6.212.62:38925
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.2
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:34:17.429335 11321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:17.430778 11321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:17.444756 11326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:17.445143 11329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:17.445447 11327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:17.448837 11321 server_base.cc:1034] running on GCE node
I20260924 11:34:17.449431 11321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:17.450423 11321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:17.451635 11321 hybrid_clock.cc:648] HybridClock initialized: now 1790249657451579 us; error 47 us; skew 500 ppm
I20260924 11:34:17.451943 11321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:34:17 dist-test-slave-cq7c krb5kdc[11082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249657, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:17.454923 11321 init.cc:377] Logged in from keytab as kudu/127.6.212.2@KRBTEST.COM (short username kudu)
I20260924 11:34:17.456223 11321 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:17.456347 11321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:17.456655 11321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:34:17.459497 11321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-1/data/instance:
uuid: "c89ab9dd536e43d5a3d842f918ab7df0"
format_stamp: "Formatted at 2026-09-24 11:34:17 on dist-test-slave-cq7c"
server_key: "126c310d0c98d49c0d31785e5969c795"
server_key_iv: "f32d9844e4f516c43868008d508d1e9c"
server_key_version: "encryptionkey@0"
I20260924 11:34:17.460297 11321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance:
uuid: "c89ab9dd536e43d5a3d842f918ab7df0"
format_stamp: "Formatted at 2026-09-24 11:34:17 on dist-test-slave-cq7c"
server_key: "126c310d0c98d49c0d31785e5969c795"
server_key_iv: "f32d9844e4f516c43868008d508d1e9c"
server_key_version: "encryptionkey@0"
I20260924 11:34:17.465281 11321 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 11:34:17.468604 11335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:17.469890 11321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:34:17.470057 11321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-1/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-1/wal
uuid: "c89ab9dd536e43d5a3d842f918ab7df0"
format_stamp: "Formatted at 2026-09-24 11:34:17 on dist-test-slave-cq7c"
server_key: "126c310d0c98d49c0d31785e5969c795"
server_key_iv: "f32d9844e4f516c43868008d508d1e9c"
server_key_version: "encryptionkey@0"
I20260924 11:34:17.470311 11321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:17.508797 11321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:17.512773 11321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:34:17.513049 11321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:17.513988 11321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:34:17.515275 11321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:34:17.515411 11321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:17.515547 11321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:34:17.515619 11321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:17.548188 11321 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.2:33523
I20260924 11:34:17.548244 11447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.2:33523 every 8 connection(s)
I20260924 11:34:17.549064 11321 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
I20260924 11:34:17.556191 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 11321
I20260924 11:34:17.556366 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance
I20260924 11:34:17.556809 6992 external_mini_cluster.cc:1428] Setting key 38461b2726b2feb6271b52747343edbf
Sep 24 11:34:17 dist-test-slave-cq7c krb5kdc[11082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249657, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.2@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:34:17.569496 11317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:17.551417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:39413 (local address 127.6.212.62:38925)
0924 11:34:17.551587 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:17.551592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:17.552773 (+ 1181us) server_negotiation.cc:407] Connection header received
0924 11:34:17.554562 (+ 1789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:17.554569 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:17.554658 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:17.554761 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:17.556513 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:17.557237 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:17.559294 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:17.559782 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:17.559913 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:17.563434 (+ 3521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:17.563440 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:17.563443 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:17.563474 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:17.566177 (+ 2703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:17.567122 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:17.567125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:17.567128 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:17.567281 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:17.567804 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:17.567807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:17.567809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:17.568046 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:17.568156 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:17.569058 (+ 902us) server_negotiation.cc:299] Negotiation successful
0924 11:34:17.569261 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 11:34:17.571148 11450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:17.551914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:38925 (local address 127.6.212.2:39413)
0924 11:34:17.552615 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:17.552656 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:17.554306 (+ 1650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:17.554974 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:17.554982 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:17.555547 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:17.556301 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:17.556327 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:17.557376 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:17.557382 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:17.559161 (+ 1779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:17.559172 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:17.559991 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:17.559998 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:17.560213 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:17.560888 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:17.560908 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:17.563172 (+ 2264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:17.566314 (+ 3142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:17.566326 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:17.566345 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:17.566981 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:17.567433 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:17.567442 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:17.567447 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:17.567695 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:17.568191 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:17.568201 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:17.568657 (+ 456us) client_negotiation.cc:769] Sending connection context
0924 11:34:17.568983 (+ 326us) client_negotiation.cc:240] Negotiation successful
0924 11:34:17.569588 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":289,"threads_started":1}
I20260924 11:34:17.572688 11448 heartbeater.cc:347] Connected to a master server at 127.6.212.62:38925
I20260924 11:34:17.573092 11448 heartbeater.cc:464] Registering TS with master...
I20260924 11:34:17.573977 11448 heartbeater.cc:511] Master 127.6.212.62:38925 requested a full tablet report, sending...
I20260924 11:34:17.575688 11126 ts_manager.cc:194] Registered new tserver with Master: c89ab9dd536e43d5a3d842f918ab7df0 (127.6.212.2:33523)
I20260924 11:34:17.576891 11126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.2@KRBTEST.COM'} at 127.6.212.2:39413
WARNING: no policy specified for kudu/127.6.212.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab.
Entry for principal kudu/127.6.212.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab.
I20260924 11:34:17.602344 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.3:0
--local_ip_for_outbound_sockets=127.6.212.3
--webserver_interface=127.6.212.3
--webserver_port=0
--tserver_master_addrs=127.6.212.62:38925
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab
--principal=kudu/127.6.212.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:44677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:34:17.931957 11453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:17.932243 11453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:17.932339 11453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:17.939581 11453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:17.939692 11453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:17.939807 11453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.3
I20260924 11:34:17.949824 11453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:44677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab
--principal=kudu/127.6.212.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.212.3
--webserver_port=0
--tserver_master_addrs=127.6.212.62:38925
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.3
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:34:17.951062 11453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:17.952366 11453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:17.964903 11458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:17.965039 11461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:17.964869 11459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:17.965811 11453 server_base.cc:1034] running on GCE node
I20260924 11:34:17.966439 11453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:17.967311 11453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:17.968546 11453 hybrid_clock.cc:648] HybridClock initialized: now 1790249657968510 us; error 26 us; skew 500 ppm
I20260924 11:34:17.968873 11453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:34:17 dist-test-slave-cq7c krb5kdc[11082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249657, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:17.971937 11453 init.cc:377] Logged in from keytab as kudu/127.6.212.3@KRBTEST.COM (short username kudu)
I20260924 11:34:17.973219 11453 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:17.973342 11453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:17.973658 11453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:34:17.976613 11453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-2/data/instance:
uuid: "44e60d6a14964213a7d364a41ad303c1"
format_stamp: "Formatted at 2026-09-24 11:34:17 on dist-test-slave-cq7c"
server_key: "09739e605e80bf76ae0a5c71f69629c5"
server_key_iv: "e2b7c22a14ddaa433292d42e511696b0"
server_key_version: "encryptionkey@0"
I20260924 11:34:17.977401 11453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance:
uuid: "44e60d6a14964213a7d364a41ad303c1"
format_stamp: "Formatted at 2026-09-24 11:34:17 on dist-test-slave-cq7c"
server_key: "09739e605e80bf76ae0a5c71f69629c5"
server_key_iv: "e2b7c22a14ddaa433292d42e511696b0"
server_key_version: "encryptionkey@0"
I20260924 11:34:17.982250 11453 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 11:34:17.985720 11467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:17.987197 11453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 11:34:17.987427 11453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-2/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-2/wal
uuid: "44e60d6a14964213a7d364a41ad303c1"
format_stamp: "Formatted at 2026-09-24 11:34:17 on dist-test-slave-cq7c"
server_key: "09739e605e80bf76ae0a5c71f69629c5"
server_key_iv: "e2b7c22a14ddaa433292d42e511696b0"
server_key_version: "encryptionkey@0"
I20260924 11:34:17.987648 11453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:18.014782 11453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:18.019181 11453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:34:18.019474 11453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:18.020448 11453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:34:18.021760 11453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:34:18.021876 11453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:18.022001 11453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:34:18.022084 11453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:18.055586 11453 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.3:39297
I20260924 11:34:18.055661 11579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.3:39297 every 8 connection(s)
I20260924 11:34:18.056406 11453 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
I20260924 11:34:18.060971 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 11453
I20260924 11:34:18.061189 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance
I20260924 11:34:18.061720 6992 external_mini_cluster.cc:1428] Setting key 2359b44a74aa955c8420765bdcbc03ef
I20260924 11:34:18.068930 11315 heartbeater.cc:503] Master 127.6.212.62:38925 was elected leader, sending a full tablet report...
Sep 24 11:34:18 dist-test-slave-cq7c krb5kdc[11082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249657, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.3@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:34:18.077069 11317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:18.058767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:55475 (local address 127.6.212.62:38925)
0924 11:34:18.058924 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:18.058929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:18.060102 (+ 1173us) server_negotiation.cc:407] Connection header received
0924 11:34:18.062450 (+ 2348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:18.062456 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:18.062541 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:18.062661 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:18.064326 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:18.065108 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:18.066642 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:18.067103 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:18.067231 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:18.070656 (+ 3425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:18.070667 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:18.070670 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:18.070702 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:18.073450 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:18.074211 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:18.074215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:18.074217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:18.074337 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:18.075100 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:18.075105 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:18.075108 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:18.075489 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:18.075638 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:18.076496 (+ 858us) server_negotiation.cc:299] Negotiation successful
0924 11:34:18.076718 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 11:34:18.078114 11582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:18.059244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:38925 (local address 127.6.212.3:55475)
0924 11:34:18.059943 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:18.059983 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:18.062154 (+ 2171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:18.062875 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:18.062883 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:18.063578 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:18.064123 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:18.064140 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:18.065273 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:18.065277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:18.066492 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:18.066503 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:18.067335 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:18.067342 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:18.067604 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:18.068217 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:18.068243 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:18.070469 (+ 2226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:18.073594 (+ 3125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:18.073601 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:18.073614 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:18.074102 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:18.074452 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:18.074461 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:18.074465 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:18.074680 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:18.075618 (+ 938us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:18.075626 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:18.076063 (+ 437us) client_negotiation.cc:769] Sending connection context
0924 11:34:18.076417 (+ 354us) client_negotiation.cc:240] Negotiation successful
0924 11:34:18.076954 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":320,"threads_started":1}
I20260924 11:34:18.079752 11580 heartbeater.cc:347] Connected to a master server at 127.6.212.62:38925
I20260924 11:34:18.080030 11580 heartbeater.cc:464] Registering TS with master...
I20260924 11:34:18.080639 11580 heartbeater.cc:511] Master 127.6.212.62:38925 requested a full tablet report, sending...
I20260924 11:34:18.081960 11126 ts_manager.cc:194] Registered new tserver with Master: 44e60d6a14964213a7d364a41ad303c1 (127.6.212.3:39297)
I20260924 11:34:18.082746 11126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.3@KRBTEST.COM'} at 127.6.212.3:55475
I20260924 11:34:18.089027 6992 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:34:18.105753 11317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:18.093458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58710 (local address 127.6.212.62:38925)
0924 11:34:18.093605 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:18.093609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:18.094084 (+ 475us) server_negotiation.cc:407] Connection header received
0924 11:34:18.094291 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:18.094295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:18.094351 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:18.094446 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:18.095917 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:18.096631 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:18.097943 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:18.098355 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:18.098486 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:18.100769 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:18.100774 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:18.100777 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:18.100806 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:18.103264 (+ 2458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:18.103890 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:18.103897 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:18.103901 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:18.104025 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:18.104387 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:18.104392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:18.104400 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:18.104692 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:18.104966 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:18.105247 (+ 281us) server_negotiation.cc:299] Negotiation successful
0924 11:34:18.105447 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 11:34:18.109340 11126 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58710:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:34:18.116400 11126 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:34:18.140341 11592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:18.127487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.2:33523 (local address 127.0.0.1:58764)
0924 11:34:18.128303 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:18.128350 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:18.128551 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:18.129726 (+ 1175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:18.129737 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:18.129777 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:34:18.130260 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:18.130276 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:18.132800 (+ 2524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:18.132808 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:18.135981 (+ 3173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:18.136004 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:18.138893 (+ 2889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:18.138902 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:18.139224 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:18.139297 (+ 73us) client_negotiation.cc:769] Sending connection context
0924 11:34:18.139482 (+ 185us) client_negotiation.cc:240] Negotiation successful
0924 11:34:18.139743 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":978,"threads_started":1}
I20260924 11:34:18.141137 11593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:18.127720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58764 (local address 127.6.212.2:33523)
0924 11:34:18.128253 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:18.128260 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:18.128449 (+ 189us) server_negotiation.cc:407] Connection header received
0924 11:34:18.128863 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:18.128870 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:18.129183 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:18.129401 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:34:18.130491 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:18.132591 (+ 2100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:18.136238 (+ 3647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:18.138248 (+ 2010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:18.138411 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:18.138727 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:18.140604 (+ 1877us) server_negotiation.cc:299] Negotiation successful
0924 11:34:18.140784 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":270,"threads_started":1}
I20260924 11:34:18.146518 11383 tablet_service.cc:1467] Processing CreateTablet for tablet e37f98451d884855b72167567dadaeca (DEFAULT_TABLE table=test-table [id=50fef27c0d734e739f91501c63f332a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:34:18.148102 11383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e37f98451d884855b72167567dadaeca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:34:18.148770 11592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:18.129266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:35479 (local address 127.0.0.1:38490)
0924 11:34:18.142222 (+ 12956us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:18.142240 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:18.142344 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:18.142932 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:18.142937 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:18.142957 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:34:18.143158 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:18.143172 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:18.144682 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:18.144687 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:18.146255 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:18.146269 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:18.148011 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:18.148017 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:18.148197 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:18.148231 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 11:34:18.148344 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 11:34:18.148479 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12844,"thread_start_us":12925,"threads_started":1}
I20260924 11:34:18.148867 11597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:18.129616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38490 (local address 127.6.212.1:35479)
0924 11:34:18.142038 (+ 12422us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:18.142048 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:18.142395 (+ 347us) server_negotiation.cc:407] Connection header received
0924 11:34:18.142525 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:18.142534 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:18.142751 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:18.142937 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:34:18.143298 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:18.144538 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:18.146456 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:18.147859 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:18.147989 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:18.148213 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:18.148384 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 11:34:18.148532 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12243,"thread_start_us":12094,"threads_started":1}
I20260924 11:34:18.150149 11595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:18.128017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.3:39297 (local address 127.0.0.1:42462)
0924 11:34:18.141783 (+ 13766us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:18.141804 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:18.141943 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:18.143033 (+ 1090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:18.143038 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:18.143063 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:34:18.143538 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:18.143551 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:18.145426 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:18.145432 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:18.147507 (+ 2075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:18.147521 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:18.149313 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:18.149322 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:18.149489 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:18.149519 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 11:34:18.149614 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 11:34:18.149761 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13589,"mutex_wait_us":531,"thread_start_us":12836,"threads_started":1}
I20260924 11:34:18.150665 11594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:18.128369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42462 (local address 127.6.212.3:39297)
0924 11:34:18.128960 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:18.128968 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:18.141927 (+ 12959us) server_negotiation.cc:407] Connection header received
0924 11:34:18.142104 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:18.142117 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:18.142423 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:18.142652 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:34:18.143777 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:18.145294 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:18.147666 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:18.149022 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:18.149154 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:18.149367 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:18.150152 (+ 785us) server_negotiation.cc:299] Negotiation successful
0924 11:34:18.150339 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":351,"threads_started":1}
I20260924 11:34:18.152837 11250 tablet_service.cc:1467] Processing CreateTablet for tablet e37f98451d884855b72167567dadaeca (DEFAULT_TABLE table=test-table [id=50fef27c0d734e739f91501c63f332a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:34:18.154284 11250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e37f98451d884855b72167567dadaeca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:34:18.153959 11515 tablet_service.cc:1467] Processing CreateTablet for tablet e37f98451d884855b72167567dadaeca (DEFAULT_TABLE table=test-table [id=50fef27c0d734e739f91501c63f332a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:34:18.155299 11515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e37f98451d884855b72167567dadaeca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:34:18.161468 11598 tablet_bootstrap.cc:492] T e37f98451d884855b72167567dadaeca P c89ab9dd536e43d5a3d842f918ab7df0: Bootstrap starting.
I20260924 11:34:18.164570 11598 tablet_bootstrap.cc:654] T e37f98451d884855b72167567dadaeca P c89ab9dd536e43d5a3d842f918ab7df0: Neither blocks nor log segments found. Creating new log.
I20260924 11:34:18.166379 11598 log.cc:824] T e37f98451d884855b72167567dadaeca P c89ab9dd536e43d5a3d842f918ab7df0: Log is configured to *not* fsync() on all Append() calls
I20260924 11:34:18.169174 11598 tablet_bootstrap.cc:492] T e37f98451d884855b72167567dadaeca P c89ab9dd536e43d5a3d842f918ab7df0: No bootstrap required, opened a new log
I20260924 11:34:18.169548 11598 ts_tablet_manager.cc:1397] T e37f98451d884855b72167567dadaeca P c89ab9dd536e43d5a3d842f918ab7df0: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260924 11:34:18.169600 11601 tablet_bootstrap.cc:492] T e37f98451d884855b72167567dadaeca P 23b0c4cf9b8045b99dda67b8839ed2d3: Bootstrap starting.
I20260924 11:34:18.171396 11600 tablet_bootstrap.cc:492] T e37f98451d884855b72167567dadaeca P 44e60d6a14964213a7d364a41ad303c1: Bootstrap starting.
I20260924 11:34:18.173211 11601 tablet_bootstrap.cc:654] T e37f98451d884855b72167567dadaeca P 23b0c4cf9b8045b99dda67b8839ed2d3: Neither blocks nor log segments found. Creating new log.
I20260924 11:34:18.174645 11601 log.cc:824] T e37f98451d884855b72167567dadaeca P 23b0c4cf9b8045b99dda67b8839ed2d3: Log is configured to *not* fsync() on all Append() calls
I20260924 11:34:18.174799 11600 tablet_bootstrap.cc:654] T e37f98451d884855b72167567dadaeca P 44e60d6a14964213a7d364a41ad303c1: Neither blocks nor log segments found. Creating new log.
I20260924 11:34:18.176088 11600 log.cc:824] T e37f98451d884855b72167567dadaeca P 44e60d6a14964213a7d364a41ad303c1: Log is configured to *not* fsync() on all Append() calls
I20260924 11:34:18.177784 11601 tablet_bootstrap.cc:492] T e37f98451d884855b72167567dadaeca P 23b0c4cf9b8045b99dda67b8839ed2d3: No bootstrap required, opened a new log
I20260924 11:34:18.178033 11601 ts_tablet_manager.cc:1397] T e37f98451d884855b72167567dadaeca P 23b0c4cf9b8045b99dda67b8839ed2d3: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260924 11:34:18.178488 11600 tablet_bootstrap.cc:492] T e37f98451d884855b72167567dadaeca P 44e60d6a14964213a7d364a41ad303c1: No bootstrap required, opened a new log
I20260924 11:34:18.178733 11600 ts_tablet_manager.cc:1397] T e37f98451d884855b72167567dadaeca P 44e60d6a14964213a7d364a41ad303c1: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 11:34:18.180711 11598 raft_consensus.cc:348] T e37f98451d884855b72167567dadaeca P c89ab9dd536e43d5a3d842f918ab7df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89ab9dd536e43d5a3d842f918ab7df0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 33523 } } peers { permanent_uuid: "23b0c4cf9b8045b99dda67b8839ed2d3" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 35479 } } peers { permanent_uuid: "44e60d6a14964213a7d364a41ad303c1" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 39297 } }
I20260924 11:34:18.181143 11598 raft_consensus.cc:374] T e37f98451d884855b72167567dadaeca P c89ab9dd536e43d5a3d842f918ab7df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:34:18.181267 11598 raft_consensus.cc:729] T e37f98451d884855b72167567dadaeca P c89ab9dd536e43d5a3d842f918ab7df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c89ab9dd536e43d5a3d842f918ab7df0, State: Initialized, Role: FOLLOWER
I20260924 11:34:18.182380 11598 consensus_queue.cc:260] T e37f98451d884855b72167567dadaeca P c89ab9dd536e43d5a3d842f918ab7df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89ab9dd536e43d5a3d842f918ab7df0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 33523 } } peers { permanent_uuid: "23b0c4cf9b8045b99dda67b8839ed2d3" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 35479 } } peers { permanent_uuid: "44e60d6a14964213a7d364a41ad303c1" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 39297 } }
I20260924 11:34:18.183501 11448 heartbeater.cc:503] Master 127.6.212.62:38925 was elected leader, sending a full tablet report...
I20260924 11:34:18.185738 11598 ts_tablet_manager.cc:1428] T e37f98451d884855b72167567dadaeca P c89ab9dd536e43d5a3d842f918ab7df0: Time spent starting tablet: real 0.016s user 0.012s sys 0.000s
I20260924 11:34:18.187346 11600 raft_consensus.cc:348] T e37f98451d884855b72167567dadaeca P 44e60d6a14964213a7d364a41ad303c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89ab9dd536e43d5a3d842f918ab7df0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 33523 } } peers { permanent_uuid: "23b0c4cf9b8045b99dda67b8839ed2d3" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 35479 } } peers { permanent_uuid: "44e60d6a14964213a7d364a41ad303c1" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 39297 } }
I20260924 11:34:18.187572 11601 raft_consensus.cc:348] T e37f98451d884855b72167567dadaeca P 23b0c4cf9b8045b99dda67b8839ed2d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89ab9dd536e43d5a3d842f918ab7df0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 33523 } } peers { permanent_uuid: "23b0c4cf9b8045b99dda67b8839ed2d3" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 35479 } } peers { permanent_uuid: "44e60d6a14964213a7d364a41ad303c1" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 39297 } }
I20260924 11:34:18.187846 11600 raft_consensus.cc:374] T e37f98451d884855b72167567dadaeca P 44e60d6a14964213a7d364a41ad303c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:34:18.187958 11600 raft_consensus.cc:729] T e37f98451d884855b72167567dadaeca P 44e60d6a14964213a7d364a41ad303c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44e60d6a14964213a7d364a41ad303c1, State: Initialized, Role: FOLLOWER
I20260924 11:34:18.187979 11601 raft_consensus.cc:374] T e37f98451d884855b72167567dadaeca P 23b0c4cf9b8045b99dda67b8839ed2d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:34:18.188071 11601 raft_consensus.cc:729] T e37f98451d884855b72167567dadaeca P 23b0c4cf9b8045b99dda67b8839ed2d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b0c4cf9b8045b99dda67b8839ed2d3, State: Initialized, Role: FOLLOWER
I20260924 11:34:18.188586 11600 consensus_queue.cc:260] T e37f98451d884855b72167567dadaeca P 44e60d6a14964213a7d364a41ad303c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89ab9dd536e43d5a3d842f918ab7df0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 33523 } } peers { permanent_uuid: "23b0c4cf9b8045b99dda67b8839ed2d3" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 35479 } } peers { permanent_uuid: "44e60d6a14964213a7d364a41ad303c1" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 39297 } }
I20260924 11:34:18.188664 11601 consensus_queue.cc:260] T e37f98451d884855b72167567dadaeca P 23b0c4cf9b8045b99dda67b8839ed2d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89ab9dd536e43d5a3d842f918ab7df0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 33523 } } peers { permanent_uuid: "23b0c4cf9b8045b99dda67b8839ed2d3" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 35479 } } peers { permanent_uuid: "44e60d6a14964213a7d364a41ad303c1" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 39297 } }
I20260924 11:34:18.189597 11580 heartbeater.cc:503] Master 127.6.212.62:38925 was elected leader, sending a full tablet report...
I20260924 11:34:18.191263 11601 ts_tablet_manager.cc:1428] T e37f98451d884855b72167567dadaeca P 23b0c4cf9b8045b99dda67b8839ed2d3: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
I20260924 11:34:18.191263 11600 ts_tablet_manager.cc:1428] T e37f98451d884855b72167567dadaeca P 44e60d6a14964213a7d364a41ad303c1: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260924 11:34:18.228175 11604 raft_consensus.cc:482] T e37f98451d884855b72167567dadaeca P c89ab9dd536e43d5a3d842f918ab7df0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:34:18.228477 11604 raft_consensus.cc:504] T e37f98451d884855b72167567dadaeca P c89ab9dd536e43d5a3d842f918ab7df0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89ab9dd536e43d5a3d842f918ab7df0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 33523 } } peers { permanent_uuid: "23b0c4cf9b8045b99dda67b8839ed2d3" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 35479 } } peers { permanent_uuid: "44e60d6a14964213a7d364a41ad303c1" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 39297 } }
I20260924 11:34:18.230634 11604 leader_election.cc:288] T e37f98451d884855b72167567dadaeca P c89ab9dd536e43d5a3d842f918ab7df0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23b0c4cf9b8045b99dda67b8839ed2d3 (127.6.212.1:35479), 44e60d6a14964213a7d364a41ad303c1 (127.6.212.3:39297)
I20260924 11:34:18.240463 11594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:18.231288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:34783 (local address 127.6.212.3:39297)
0924 11:34:18.231551 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:18.231561 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:18.232132 (+ 571us) server_negotiation.cc:407] Connection header received
0924 11:34:18.232336 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:18.232342 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:18.232420 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:18.232522 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:34:18.233658 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:18.234751 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:18.237972 (+ 3221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:18.239205 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:18.239324 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:18.239454 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:18.240045 (+ 591us) server_negotiation.cc:299] Negotiation successful
0924 11:34:18.240199 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260924 11:34:18.240540 11597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:18.231145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:45875 (local address 127.6.212.1:35479)
0924 11:34:18.231420 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:18.231429 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:18.232135 (+ 706us) server_negotiation.cc:407] Connection header received
0924 11:34:18.232351 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:18.232355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:18.232420 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:18.232522 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:34:18.233670 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:18.234751 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:18.237972 (+ 3221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:18.239179 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:18.239324 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:18.239454 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:18.240071 (+ 617us) server_negotiation.cc:299] Negotiation successful
0924 11:34:18.240263 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 11:34:18.241295 11608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:18.230938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:35479 (local address 127.6.212.2:45875)
0924 11:34:18.232031 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:18.232050 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:18.232223 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:18.232650 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:18.232656 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:18.232703 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:34:18.233510 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:18.233519 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:18.234902 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:18.234907 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:18.237806 (+ 2899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:18.237823 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:18.239538 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:18.239545 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:18.239841 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:18.239887 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 11:34:18.240719 (+ 832us) client_negotiation.cc:240] Negotiation successful
0924 11:34:18.240845 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":977,"mutex_wait_us":226,"spinlock_wait_cycles":40960,"thread_start_us":816,"threads_started":1}
I20260924 11:34:18.241534 11607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:18.230875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.3:39297 (local address 127.6.212.2:34783)
0924 11:34:18.231972 (+ 1097us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:18.231993 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:18.232179 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:18.232650 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:18.232662 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:18.232712 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:34:18.233474 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:18.233495 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:18.234901 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:18.234907 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:18.237815 (+ 2908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:18.237824 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:18.239503 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:18.239511 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:18.239750 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:18.239883 (+ 133us) client_negotiation.cc:769] Sending connection context
0924 11:34:18.239993 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 11:34:18.241061 (+ 1068us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":884,"mutex_wait_us":256,"spinlock_wait_cycles":16384,"thread_start_us":565,"threads_started":1}
I20260924 11:34:18.242414 11270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e37f98451d884855b72167567dadaeca" candidate_uuid: "c89ab9dd536e43d5a3d842f918ab7df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b0c4cf9b8045b99dda67b8839ed2d3" is_pre_election: true
I20260924 11:34:18.242482 11535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e37f98451d884855b72167567dadaeca" candidate_uuid: "c89ab9dd536e43d5a3d842f918ab7df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44e60d6a14964213a7d364a41ad303c1" is_pre_election: true
I20260924 11:34:18.242950 11535 raft_consensus.cc:2454] T e37f98451d884855b72167567dadaeca P 44e60d6a14964213a7d364a41ad303c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c89ab9dd536e43d5a3d842f918ab7df0 in term 0.
I20260924 11:34:18.242954 11270 raft_consensus.cc:2454] T e37f98451d884855b72167567dadaeca P 23b0c4cf9b8045b99dda67b8839ed2d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c89ab9dd536e43d5a3d842f918ab7df0 in term 0.
I20260924 11:34:18.243654 11336 leader_election.cc:302] T e37f98451d884855b72167567dadaeca P c89ab9dd536e43d5a3d842f918ab7df0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 23b0c4cf9b8045b99dda67b8839ed2d3, 44e60d6a14964213a7d364a41ad303c1, c89ab9dd536e43d5a3d842f918ab7df0; no voters:
I20260924 11:34:18.244047 11604 raft_consensus.cc:2781] T e37f98451d884855b72167567dadaeca P c89ab9dd536e43d5a3d842f918ab7df0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:34:18.244192 11604 raft_consensus.cc:482] T e37f98451d884855b72167567dadaeca P c89ab9dd536e43d5a3d842f918ab7df0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:34:18.244319 11604 raft_consensus.cc:3037] T e37f98451d884855b72167567dadaeca P c89ab9dd536e43d5a3d842f918ab7df0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:34:18.246438 11604 raft_consensus.cc:504] T e37f98451d884855b72167567dadaeca P c89ab9dd536e43d5a3d842f918ab7df0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89ab9dd536e43d5a3d842f918ab7df0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 33523 } } peers { permanent_uuid: "23b0c4cf9b8045b99dda67b8839ed2d3" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 35479 } } peers { permanent_uuid: "44e60d6a14964213a7d364a41ad303c1" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 39297 } }
I20260924 11:34:18.247027 11604 leader_election.cc:288] T e37f98451d884855b72167567dadaeca P c89ab9dd536e43d5a3d842f918ab7df0 [CANDIDATE]: Term 1 election: Requested vote from peers 23b0c4cf9b8045b99dda67b8839ed2d3 (127.6.212.1:35479), 44e60d6a14964213a7d364a41ad303c1 (127.6.212.3:39297)
I20260924 11:34:18.247525 11270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e37f98451d884855b72167567dadaeca" candidate_uuid: "c89ab9dd536e43d5a3d842f918ab7df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b0c4cf9b8045b99dda67b8839ed2d3"
I20260924 11:34:18.247601 11535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e37f98451d884855b72167567dadaeca" candidate_uuid: "c89ab9dd536e43d5a3d842f918ab7df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44e60d6a14964213a7d364a41ad303c1"
I20260924 11:34:18.247720 11270 raft_consensus.cc:3037] T e37f98451d884855b72167567dadaeca P 23b0c4cf9b8045b99dda67b8839ed2d3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:34:18.247836 11535 raft_consensus.cc:3037] T e37f98451d884855b72167567dadaeca P 44e60d6a14964213a7d364a41ad303c1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:34:18.249843 11270 raft_consensus.cc:2454] T e37f98451d884855b72167567dadaeca P 23b0c4cf9b8045b99dda67b8839ed2d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c89ab9dd536e43d5a3d842f918ab7df0 in term 1.
I20260924 11:34:18.250169 11535 raft_consensus.cc:2454] T e37f98451d884855b72167567dadaeca P 44e60d6a14964213a7d364a41ad303c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c89ab9dd536e43d5a3d842f918ab7df0 in term 1.
I20260924 11:34:18.250224 11336 leader_election.cc:302] T e37f98451d884855b72167567dadaeca P c89ab9dd536e43d5a3d842f918ab7df0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23b0c4cf9b8045b99dda67b8839ed2d3, c89ab9dd536e43d5a3d842f918ab7df0; no voters:
I20260924 11:34:18.250538 11604 raft_consensus.cc:2781] T e37f98451d884855b72167567dadaeca P c89ab9dd536e43d5a3d842f918ab7df0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:34:18.251008 11604 raft_consensus.cc:686] T e37f98451d884855b72167567dadaeca P c89ab9dd536e43d5a3d842f918ab7df0 [term 1 LEADER]: Becoming Leader. State: Replica: c89ab9dd536e43d5a3d842f918ab7df0, State: Running, Role: LEADER
I20260924 11:34:18.251595 11604 consensus_queue.cc:237] T e37f98451d884855b72167567dadaeca P c89ab9dd536e43d5a3d842f918ab7df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89ab9dd536e43d5a3d842f918ab7df0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 33523 } } peers { permanent_uuid: "23b0c4cf9b8045b99dda67b8839ed2d3" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 35479 } } peers { permanent_uuid: "44e60d6a14964213a7d364a41ad303c1" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 39297 } }
I20260924 11:34:18.256950 11125 catalog_manager.cc:5495] T e37f98451d884855b72167567dadaeca P c89ab9dd536e43d5a3d842f918ab7df0 reported cstate change: term changed from 0 to 1, leader changed from <none> to c89ab9dd536e43d5a3d842f918ab7df0 (127.6.212.2). New cstate: current_term: 1 leader_uuid: "c89ab9dd536e43d5a3d842f918ab7df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89ab9dd536e43d5a3d842f918ab7df0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 33523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23b0c4cf9b8045b99dda67b8839ed2d3" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 35479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44e60d6a14964213a7d364a41ad303c1" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 39297 } health_report { overall_health: UNKNOWN } } }
I20260924 11:34:18.276787 11593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:18.271213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58772 (local address 127.6.212.2:33523)
0924 11:34:18.271426 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:18.271434 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:18.271589 (+ 155us) server_negotiation.cc:407] Connection header received
0924 11:34:18.271781 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:18.271784 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:18.271857 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:18.271978 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:34:18.272349 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:18.273235 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:18.274765 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:18.275140 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:18.275239 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:18.275728 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:34:18.276226 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:34:18.276333 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:18.276429 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 11:34:18.276491 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 11:34:18.284777 11535 raft_consensus.cc:1261] T e37f98451d884855b72167567dadaeca P 44e60d6a14964213a7d364a41ad303c1 [term 1 FOLLOWER]: Refusing update from remote peer c89ab9dd536e43d5a3d842f918ab7df0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:34:18.284776 11270 raft_consensus.cc:1261] T e37f98451d884855b72167567dadaeca P 23b0c4cf9b8045b99dda67b8839ed2d3 [term 1 FOLLOWER]: Refusing update from remote peer c89ab9dd536e43d5a3d842f918ab7df0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:34:18.285705 11604 consensus_queue.cc:1035] T e37f98451d884855b72167567dadaeca P c89ab9dd536e43d5a3d842f918ab7df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44e60d6a14964213a7d364a41ad303c1" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 39297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:34:18.285996 11609 consensus_queue.cc:1035] T e37f98451d884855b72167567dadaeca P c89ab9dd536e43d5a3d842f918ab7df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23b0c4cf9b8045b99dda67b8839ed2d3" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 35479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 11:34:18.288123 11316 tablet.cc:2369] T e37f98451d884855b72167567dadaeca P 23b0c4cf9b8045b99dda67b8839ed2d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:34:18.301581 11614 mvcc.cc:205] Tried to move back new op lower bound from 7332862600324784128 to 7332862600202813440. Current Snapshot: MvccSnapshot[applied={T|T < 7332862600324784128}]
I20260924 11:34:18.312039 11126 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:34:18.312326 11126 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:34:18.315620 11126 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d81c08e1dfa43b19f2532ad89b969e9: Sending DeleteTablet for 3 replicas of tablet e37f98451d884855b72167567dadaeca
I20260924 11:34:18.316679 11383 tablet_service.cc:1514] Processing DeleteTablet for tablet e37f98451d884855b72167567dadaeca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:34:18 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:58764
I20260924 11:34:18.316888 11515 tablet_service.cc:1514] Processing DeleteTablet for tablet e37f98451d884855b72167567dadaeca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:34:18 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:42462
I20260924 11:34:18.316926 11250 tablet_service.cc:1514] Processing DeleteTablet for tablet e37f98451d884855b72167567dadaeca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:34:18 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:38490
I20260924 11:34:18.320073 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 11188
I20260924 11:34:18.332073 11623 tablet_replica.cc:330] T e37f98451d884855b72167567dadaeca P 44e60d6a14964213a7d364a41ad303c1: stopping tablet replica
I20260924 11:34:18.332726 11623 raft_consensus.cc:2229] T e37f98451d884855b72167567dadaeca P 44e60d6a14964213a7d364a41ad303c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:34:18.333402 11623 raft_consensus.cc:2258] T e37f98451d884855b72167567dadaeca P 44e60d6a14964213a7d364a41ad303c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:34:18.336068 11623 ts_tablet_manager.cc:1905] T e37f98451d884855b72167567dadaeca P 44e60d6a14964213a7d364a41ad303c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:34:18.344807 11623 ts_tablet_manager.cc:1918] T e37f98451d884855b72167567dadaeca P 44e60d6a14964213a7d364a41ad303c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:34:18.345052 11623 log.cc:1196] T e37f98451d884855b72167567dadaeca P 44e60d6a14964213a7d364a41ad303c1: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-2/wal/wals/e37f98451d884855b72167567dadaeca
I20260924 11:34:18.345682 11623 ts_tablet_manager.cc:1939] T e37f98451d884855b72167567dadaeca P 44e60d6a14964213a7d364a41ad303c1: Deleting consensus metadata
I20260924 11:34:18.347721 11112 catalog_manager.cc:4841] TS 44e60d6a14964213a7d364a41ad303c1 (127.6.212.3:39297): tablet e37f98451d884855b72167567dadaeca (table test-table [id=50fef27c0d734e739f91501c63f332a7]) successfully deleted
I20260924 11:34:18.349689 11624 tablet_replica.cc:330] T e37f98451d884855b72167567dadaeca P 23b0c4cf9b8045b99dda67b8839ed2d3: stopping tablet replica
I20260924 11:34:18.350231 11624 raft_consensus.cc:2229] T e37f98451d884855b72167567dadaeca P 23b0c4cf9b8045b99dda67b8839ed2d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:34:18.350836 11624 raft_consensus.cc:2258] T e37f98451d884855b72167567dadaeca P 23b0c4cf9b8045b99dda67b8839ed2d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:34:18.353255 11624 ts_tablet_manager.cc:1905] T e37f98451d884855b72167567dadaeca P 23b0c4cf9b8045b99dda67b8839ed2d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:34:18.353873 11622 tablet_replica.cc:330] T e37f98451d884855b72167567dadaeca P c89ab9dd536e43d5a3d842f918ab7df0: stopping tablet replica
I20260924 11:34:18.354321 11622 raft_consensus.cc:2229] T e37f98451d884855b72167567dadaeca P c89ab9dd536e43d5a3d842f918ab7df0 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:34:18.355062 11622 raft_consensus.cc:2258] T e37f98451d884855b72167567dadaeca P c89ab9dd536e43d5a3d842f918ab7df0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:34:18.356881 11622 ts_tablet_manager.cc:1905] T e37f98451d884855b72167567dadaeca P c89ab9dd536e43d5a3d842f918ab7df0: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 11:34:18 dist-test-slave-cq7c krb5kdc[11082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.1@KRBTEST.COM
I20260924 11:34:18.361682 11624 ts_tablet_manager.cc:1918] T e37f98451d884855b72167567dadaeca P 23b0c4cf9b8045b99dda67b8839ed2d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:34:18.361913 11624 log.cc:1196] T e37f98451d884855b72167567dadaeca P 23b0c4cf9b8045b99dda67b8839ed2d3: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-0/wal/wals/e37f98451d884855b72167567dadaeca
I20260924 11:34:18.362525 11624 ts_tablet_manager.cc:1939] T e37f98451d884855b72167567dadaeca P 23b0c4cf9b8045b99dda67b8839ed2d3: Deleting consensus metadata
I20260924 11:34:18.364414 11110 catalog_manager.cc:4841] TS 23b0c4cf9b8045b99dda67b8839ed2d3 (127.6.212.1:35479): tablet e37f98451d884855b72167567dadaeca (table test-table [id=50fef27c0d734e739f91501c63f332a7]) successfully deleted
I20260924 11:34:18.364873 11622 ts_tablet_manager.cc:1918] T e37f98451d884855b72167567dadaeca P c89ab9dd536e43d5a3d842f918ab7df0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:34:18.365113 11622 log.cc:1196] T e37f98451d884855b72167567dadaeca P c89ab9dd536e43d5a3d842f918ab7df0: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790249617057452-6992-0/minicluster-data/ts-1/wal/wals/e37f98451d884855b72167567dadaeca
I20260924 11:34:18.365756 11622 ts_tablet_manager.cc:1939] T e37f98451d884855b72167567dadaeca P c89ab9dd536e43d5a3d842f918ab7df0: Deleting consensus metadata
I20260924 11:34:18.366166 11597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:18.321786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38496 (local address 127.6.212.1:35479)
0924 11:34:18.322015 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:18.322024 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:18.346631 (+ 24607us) server_negotiation.cc:407] Connection header received
0924 11:34:18.347140 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:18.347146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:18.347241 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:18.347321 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:18.349216 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:18.350417 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:18.352293 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:18.352871 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:18.352972 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:18.358458 (+ 5486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:18.358471 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:18.358488 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:18.358531 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:18.362584 (+ 4053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:18.363474 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:18.363485 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:18.363490 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:18.363661 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:18.364077 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:18.364082 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:18.364086 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:18.364541 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:18.364855 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:18.365307 (+ 452us) server_negotiation.cc:299] Negotiation successful
0924 11:34:18.365547 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 11:34:18.366856 11310 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:34:18.367394 11110 catalog_manager.cc:4841] TS c89ab9dd536e43d5a3d842f918ab7df0 (127.6.212.2:33523): tablet e37f98451d884855b72167567dadaeca (table test-table [id=50fef27c0d734e739f91501c63f332a7]) successfully deleted
I20260924 11:34:18.457134 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 11188
I20260924 11:34:18.480059 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 11321
Sep 24 11:34:18 dist-test-slave-cq7c krb5kdc[11082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.2@KRBTEST.COM
I20260924 11:34:18.495780 11593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:18.480819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58774 (local address 127.6.212.2:33523)
0924 11:34:18.481024 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:18.481030 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:18.481053 (+ 23us) server_negotiation.cc:407] Connection header received
0924 11:34:18.481170 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:18.481176 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:18.481261 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:18.481376 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:18.482506 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:18.483822 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:18.485105 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:18.485593 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:18.485729 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:18.489996 (+ 4267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:18.490007 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:18.490023 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:18.490055 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:18.493126 (+ 3071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:18.493755 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:18.493762 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:18.493764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:18.493905 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:18.494260 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:18.494266 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:18.494269 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:18.494625 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:18.494962 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:18.495184 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 11:34:18.495418 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 11:34:18.496608 11443 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:34:18.607479 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 11321
I20260924 11:34:18.625558 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 11453
Sep 24 11:34:18 dist-test-slave-cq7c krb5kdc[11082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.3@KRBTEST.COM
I20260924 11:34:18.640049 11594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:18.626242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42464 (local address 127.6.212.3:39297)
0924 11:34:18.626439 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:18.626443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:18.626461 (+ 18us) server_negotiation.cc:407] Connection header received
0924 11:34:18.626545 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:18.626549 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:18.626614 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:18.626702 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:18.627993 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:18.628722 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:18.629887 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:18.630275 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:18.630364 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:18.634767 (+ 4403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:18.634780 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:18.634796 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:18.634829 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:18.637331 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:18.637960 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:18.637969 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:18.637971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:18.638101 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:18.638452 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:18.638457 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:18.638464 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:18.638813 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:18.639098 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:18.639550 (+ 452us) server_negotiation.cc:299] Negotiation successful
0924 11:34:18.639781 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 11:34:18.640704 11575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:34:18.728451 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 11453
I20260924 11:34:18.744990 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 11095
I20260924 11:34:18.746086 11156 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:34:18.852731 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 11095
2026-09-24T11:34:18Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4129 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T11:34:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:34:18Z Disabled control of system clock
I20260924 11:34:18.920271 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:35961
--webserver_interface=127.6.212.62
--webserver_port=0
--builtin_ntp_servers=127.6.212.20:46649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:35961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 11:34:19.244822 11633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:19.245535 11633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:19.245682 11633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:19.258541 11633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:34:19.258684 11633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:19.258750 11633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:19.258848 11633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:34:19.258913 11633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:34:19.272850 11633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:46649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:35961
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:35961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.212.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:34:19.274471 11633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:19.276666 11633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:19.290164 11638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:19.290289 11641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:19.290130 11639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:19.291216 11633 server_base.cc:1034] running on GCE node
I20260924 11:34:19.292007 11633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:19.293346 11633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:19.294580 11633 hybrid_clock.cc:648] HybridClock initialized: now 1790249659294536 us; error 36 us; skew 500 ppm
I20260924 11:34:19.295001 11633 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:34:19.297102 11633 webserver.cc:460] Webserver started at http://127.6.212.62:40567/ using document root <none> and password file <none>
I20260924 11:34:19.297762 11633 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:19.297881 11633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:19.298200 11633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:34:19.300979 11633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "09bcc0e367ee44a9bcaf3fb6d8d1d1c6"
format_stamp: "Formatted at 2026-09-24 11:34:19 on dist-test-slave-cq7c"
server_key: "404b09557b1fdca99522169924a33262"
server_key_iv: "c674113995b9bc2a9ae9140f76289c44"
server_key_version: "encryptionkey@0"
I20260924 11:34:19.301733 11633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "09bcc0e367ee44a9bcaf3fb6d8d1d1c6"
format_stamp: "Formatted at 2026-09-24 11:34:19 on dist-test-slave-cq7c"
server_key: "404b09557b1fdca99522169924a33262"
server_key_iv: "c674113995b9bc2a9ae9140f76289c44"
server_key_version: "encryptionkey@0"
I20260924 11:34:19.306447 11633 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260924 11:34:19.309597 11647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:19.311105 11633 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 11:34:19.311304 11633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "09bcc0e367ee44a9bcaf3fb6d8d1d1c6"
format_stamp: "Formatted at 2026-09-24 11:34:19 on dist-test-slave-cq7c"
server_key: "404b09557b1fdca99522169924a33262"
server_key_iv: "c674113995b9bc2a9ae9140f76289c44"
server_key_version: "encryptionkey@0"
I20260924 11:34:19.311582 11633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:19.317266 11633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:19.318079 11633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:34:19.318389 11633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:19.338063 11633 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.62:35961
I20260924 11:34:19.338093 11698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.62:35961 every 8 connection(s)
I20260924 11:34:19.339391 11633 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:34:19.343321 11699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:34:19.347788 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 11633
I20260924 11:34:19.347992 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:34:19.348894 6992 external_mini_cluster.cc:1428] Setting key 6a61237f5135f683bf083cb30e891848
I20260924 11:34:19.356122 11699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09bcc0e367ee44a9bcaf3fb6d8d1d1c6: Bootstrap starting.
I20260924 11:34:19.358316 11702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:19.351136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60894 (local address 127.6.212.62:35961)
0924 11:34:19.352094 (+ 958us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:19.352104 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:19.352142 (+ 38us) server_negotiation.cc:407] Connection header received
0924 11:34:19.354424 (+ 2282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:19.354445 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:19.354823 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:19.355270 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:19.356407 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:19.356412 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:19.356424 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:34:19.356444 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:19.356495 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:34:19.356512 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:19.356746 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:19.356876 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 11:34:19.357108 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":376,"threads_started":1}
I20260924 11:34:19.359803 11699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09bcc0e367ee44a9bcaf3fb6d8d1d1c6: Neither blocks nor log segments found. Creating new log.
I20260924 11:34:19.360841 11699 log.cc:824] T 00000000000000000000000000000000 P 09bcc0e367ee44a9bcaf3fb6d8d1d1c6: Log is configured to *not* fsync() on all Append() calls
I20260924 11:34:19.363555 11699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09bcc0e367ee44a9bcaf3fb6d8d1d1c6: No bootstrap required, opened a new log
I20260924 11:34:19.371712 11699 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09bcc0e367ee44a9bcaf3fb6d8d1d1c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09bcc0e367ee44a9bcaf3fb6d8d1d1c6" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 35961 } }
I20260924 11:34:19.372109 11699 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09bcc0e367ee44a9bcaf3fb6d8d1d1c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:34:19.372237 11699 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09bcc0e367ee44a9bcaf3fb6d8d1d1c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09bcc0e367ee44a9bcaf3fb6d8d1d1c6, State: Initialized, Role: FOLLOWER
I20260924 11:34:19.372864 11699 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09bcc0e367ee44a9bcaf3fb6d8d1d1c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09bcc0e367ee44a9bcaf3fb6d8d1d1c6" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 35961 } }
I20260924 11:34:19.373131 11699 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09bcc0e367ee44a9bcaf3fb6d8d1d1c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:34:19.373303 11699 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09bcc0e367ee44a9bcaf3fb6d8d1d1c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:34:19.373486 11699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09bcc0e367ee44a9bcaf3fb6d8d1d1c6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:34:19.376199 11699 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09bcc0e367ee44a9bcaf3fb6d8d1d1c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09bcc0e367ee44a9bcaf3fb6d8d1d1c6" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 35961 } }
I20260924 11:34:19.376740 11699 leader_election.cc:302] T 00000000000000000000000000000000 P 09bcc0e367ee44a9bcaf3fb6d8d1d1c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09bcc0e367ee44a9bcaf3fb6d8d1d1c6; no voters:
I20260924 11:34:19.377347 11699 leader_election.cc:288] T 00000000000000000000000000000000 P 09bcc0e367ee44a9bcaf3fb6d8d1d1c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:34:19.377511 11704 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09bcc0e367ee44a9bcaf3fb6d8d1d1c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:34:19.378232 11704 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09bcc0e367ee44a9bcaf3fb6d8d1d1c6 [term 1 LEADER]: Becoming Leader. State: Replica: 09bcc0e367ee44a9bcaf3fb6d8d1d1c6, State: Running, Role: LEADER
I20260924 11:34:19.378763 11704 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09bcc0e367ee44a9bcaf3fb6d8d1d1c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09bcc0e367ee44a9bcaf3fb6d8d1d1c6" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 35961 } }
I20260924 11:34:19.379706 11699 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09bcc0e367ee44a9bcaf3fb6d8d1d1c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:34:19.381609 11705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09bcc0e367ee44a9bcaf3fb6d8d1d1c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09bcc0e367ee44a9bcaf3fb6d8d1d1c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09bcc0e367ee44a9bcaf3fb6d8d1d1c6" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 35961 } } }
I20260924 11:34:19.381783 11706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09bcc0e367ee44a9bcaf3fb6d8d1d1c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09bcc0e367ee44a9bcaf3fb6d8d1d1c6. Latest consensus state: current_term: 1 leader_uuid: "09bcc0e367ee44a9bcaf3fb6d8d1d1c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09bcc0e367ee44a9bcaf3fb6d8d1d1c6" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 35961 } } }
I20260924 11:34:19.382411 11705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09bcc0e367ee44a9bcaf3fb6d8d1d1c6 [sys.catalog]: This master's current role is: LEADER
I20260924 11:34:19.382799 11706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09bcc0e367ee44a9bcaf3fb6d8d1d1c6 [sys.catalog]: This master's current role is: LEADER
I20260924 11:34:19.383977 11712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:34:19.389026 11712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:34:19.398092 11712 catalog_manager.cc:1347] Generated new cluster ID: a5d6a07a149a44978db08b5a109cda9a
I20260924 11:34:19.398214 11712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:34:19.416837 11712 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:34:19.417423 11712 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:34:19.430455 11712 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09bcc0e367ee44a9bcaf3fb6d8d1d1c6: Generated new TSK 0
I20260924 11:34:19.431623 11712 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:34:19.443495 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.1:0
--local_ip_for_outbound_sockets=127.6.212.1
--webserver_interface=127.6.212.1
--webserver_port=0
--tserver_master_addrs=127.6.212.62:35961
--builtin_ntp_servers=127.6.212.20:46649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 11:34:19.764197 11723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:19.764467 11723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:19.764638 11723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:19.771854 11723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:19.771951 11723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:19.772126 11723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.1
I20260924 11:34:19.781220 11723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:46649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.212.1
--webserver_port=0
--tserver_master_addrs=127.6.212.62:35961
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.1
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:34:19.782573 11723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:19.784565 11723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:19.797830 11728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:19.797919 11731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:19.797889 11729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:19.798336 11723 server_base.cc:1034] running on GCE node
I20260924 11:34:19.799067 11723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:19.800091 11723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:19.801321 11723 hybrid_clock.cc:648] HybridClock initialized: now 1790249659801276 us; error 60 us; skew 500 ppm
I20260924 11:34:19.801630 11723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:34:19.803920 11723 webserver.cc:460] Webserver started at http://127.6.212.1:46567/ using document root <none> and password file <none>
I20260924 11:34:19.804593 11723 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:19.804728 11723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:19.805037 11723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:34:19.807871 11723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-0/data/instance:
uuid: "26e38333eacd4d8a829d25c959e84f25"
format_stamp: "Formatted at 2026-09-24 11:34:19 on dist-test-slave-cq7c"
server_key: "e47d48c15ed93d271572053731e3f94a"
server_key_iv: "1bf75a25d40e5f02279aadd1f153b283"
server_key_version: "encryptionkey@0"
I20260924 11:34:19.808622 11723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance:
uuid: "26e38333eacd4d8a829d25c959e84f25"
format_stamp: "Formatted at 2026-09-24 11:34:19 on dist-test-slave-cq7c"
server_key: "e47d48c15ed93d271572053731e3f94a"
server_key_iv: "1bf75a25d40e5f02279aadd1f153b283"
server_key_version: "encryptionkey@0"
I20260924 11:34:19.813588 11723 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 11:34:19.816987 11737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:19.818207 11723 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 11:34:19.818373 11723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-0/wal
uuid: "26e38333eacd4d8a829d25c959e84f25"
format_stamp: "Formatted at 2026-09-24 11:34:19 on dist-test-slave-cq7c"
server_key: "e47d48c15ed93d271572053731e3f94a"
server_key_iv: "1bf75a25d40e5f02279aadd1f153b283"
server_key_version: "encryptionkey@0"
I20260924 11:34:19.818600 11723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:19.824267 11723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:19.825001 11723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:34:19.825258 11723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:19.826124 11723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:34:19.827569 11723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:34:19.827699 11723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:19.827823 11723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:34:19.827917 11723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:19.861002 11723 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.1:39603
I20260924 11:34:19.861048 11849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.1:39603 every 8 connection(s)
I20260924 11:34:19.862377 11723 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
I20260924 11:34:19.870539 11852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:19.865073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:52845 (local address 127.6.212.62:35961)
0924 11:34:19.865484 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:19.865489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:19.866796 (+ 1307us) server_negotiation.cc:407] Connection header received
0924 11:34:19.868583 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:19.868587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:19.868644 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:19.868747 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:19.869925 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:19.869929 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:19.869932 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:34:19.869948 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:19.869987 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:34:19.869998 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:19.870099 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:19.870288 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 11:34:19.870337 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":218,"threads_started":1}
I20260924 11:34:19.871495 11853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:19.865696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35961 (local address 127.6.212.1:52845)
0924 11:34:19.866638 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:19.866677 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:19.868328 (+ 1651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:19.868944 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:19.868952 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:19.869047 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:19.869655 (+ 608us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:34:19.869657 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:19.869693 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:34:19.869771 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:19.870098 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:19.870106 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:19.870112 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 11:34:19.870316 (+ 204us) client_negotiation.cc:240] Negotiation successful
0924 11:34:19.870505 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":287,"threads_started":1}
I20260924 11:34:19.872115 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 11723
I20260924 11:34:19.872325 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance
I20260924 11:34:19.872924 6992 external_mini_cluster.cc:1428] Setting key ce5762eb74f3170d3f582f1d1bc9d360
I20260924 11:34:19.873097 11850 heartbeater.cc:347] Connected to a master server at 127.6.212.62:35961
I20260924 11:34:19.873402 11850 heartbeater.cc:464] Registering TS with master...
I20260924 11:34:19.874066 11850 heartbeater.cc:511] Master 127.6.212.62:35961 requested a full tablet report, sending...
I20260924 11:34:19.876057 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.2:0
--local_ip_for_outbound_sockets=127.6.212.2
--webserver_interface=127.6.212.2
--webserver_port=0
--tserver_master_addrs=127.6.212.62:35961
--builtin_ntp_servers=127.6.212.20:46649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 11:34:19.876335 11664 ts_manager.cc:194] Registered new tserver with Master: 26e38333eacd4d8a829d25c959e84f25 (127.6.212.1:39603)
W20260924 11:34:20.202073 11854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:20.202347 11854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:20.202446 11854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:20.209631 11854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:20.209745 11854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:20.209872 11854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.2
I20260924 11:34:20.218989 11854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:46649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.212.2
--webserver_port=0
--tserver_master_addrs=127.6.212.62:35961
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.2
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:34:20.220306 11854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:20.222077 11854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:20.235328 11862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:20.235437 11859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:20.235805 11860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:20.240913 11854 server_base.cc:1034] running on GCE node
I20260924 11:34:20.241575 11854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:20.242609 11854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:20.243816 11854 hybrid_clock.cc:648] HybridClock initialized: now 1790249660243748 us; error 56 us; skew 500 ppm
I20260924 11:34:20.244158 11854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:34:20.246330 11854 webserver.cc:460] Webserver started at http://127.6.212.2:43017/ using document root <none> and password file <none>
I20260924 11:34:20.247064 11854 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:20.247180 11854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:20.247555 11854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:34:20.250385 11854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-1/data/instance:
uuid: "e1fcad3c1d23477dbdb666fcfbb81f78"
format_stamp: "Formatted at 2026-09-24 11:34:20 on dist-test-slave-cq7c"
server_key: "c8910d04754050352ece0560ebb30323"
server_key_iv: "580e4874fb3b910367296f0f2ae00594"
server_key_version: "encryptionkey@0"
I20260924 11:34:20.251143 11854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance:
uuid: "e1fcad3c1d23477dbdb666fcfbb81f78"
format_stamp: "Formatted at 2026-09-24 11:34:20 on dist-test-slave-cq7c"
server_key: "c8910d04754050352ece0560ebb30323"
server_key_iv: "580e4874fb3b910367296f0f2ae00594"
server_key_version: "encryptionkey@0"
I20260924 11:34:20.256235 11854 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260924 11:34:20.259519 11868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:20.261412 11854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 11:34:20.261596 11854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-1/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-1/wal
uuid: "e1fcad3c1d23477dbdb666fcfbb81f78"
format_stamp: "Formatted at 2026-09-24 11:34:20 on dist-test-slave-cq7c"
server_key: "c8910d04754050352ece0560ebb30323"
server_key_iv: "580e4874fb3b910367296f0f2ae00594"
server_key_version: "encryptionkey@0"
I20260924 11:34:20.261868 11854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:20.267935 11854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:20.268689 11854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:34:20.268954 11854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:20.269833 11854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:34:20.271148 11854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:34:20.271265 11854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:20.271402 11854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:34:20.271502 11854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260924 11:34:20.304005 11854 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.2:39309
I20260924 11:34:20.304091 11980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.2:39309 every 8 connection(s)
I20260924 11:34:20.305318 11854 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
I20260924 11:34:20.306006 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 11854
I20260924 11:34:20.306257 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance
I20260924 11:34:20.306871 6992 external_mini_cluster.cc:1428] Setting key e2bb272e5f6a7a1f04e42f4ac1992909
I20260924 11:34:20.310145 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.3:0
--local_ip_for_outbound_sockets=127.6.212.3
--webserver_interface=127.6.212.3
--webserver_port=0
--tserver_master_addrs=127.6.212.62:35961
--builtin_ntp_servers=127.6.212.20:46649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 11:34:20.313992 11852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:20.308270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:52727 (local address 127.6.212.62:35961)
0924 11:34:20.308454 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:20.308460 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:20.310029 (+ 1569us) server_negotiation.cc:407] Connection header received
0924 11:34:20.311772 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:20.311777 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:20.311850 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:20.311944 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:20.313276 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:20.313281 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:20.313286 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:34:20.313309 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:20.313337 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:34:20.313348 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:20.313461 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:20.313646 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 11:34:20.313719 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 11:34:20.315054 11983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:20.308851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35961 (local address 127.6.212.2:52727)
0924 11:34:20.309889 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:20.309925 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:20.311524 (+ 1599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:20.312176 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:20.312185 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:20.312277 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:20.312970 (+ 693us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:34:20.312973 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:20.313009 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:34:20.313092 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:20.313437 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:20.313446 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:20.313454 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 11:34:20.313761 (+ 307us) client_negotiation.cc:240] Negotiation successful
0924 11:34:20.314014 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":366,"threads_started":1}
I20260924 11:34:20.316453 11981 heartbeater.cc:347] Connected to a master server at 127.6.212.62:35961
I20260924 11:34:20.316758 11981 heartbeater.cc:464] Registering TS with master...
I20260924 11:34:20.317327 11981 heartbeater.cc:511] Master 127.6.212.62:35961 requested a full tablet report, sending...
I20260924 11:34:20.318674 11664 ts_manager.cc:194] Registered new tserver with Master: e1fcad3c1d23477dbdb666fcfbb81f78 (127.6.212.2:39309)
W20260924 11:34:20.633466 11984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:20.633809 11984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:20.633976 11984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:20.641269 11984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:20.641400 11984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:20.641579 11984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.3
I20260924 11:34:20.650496 11984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:46649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.212.3
--webserver_port=0
--tserver_master_addrs=127.6.212.62:35961
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.3
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:34:20.651777 11984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:20.653590 11984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:20.666389 11989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:20.666435 11992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:20.666339 11990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:20.667246 11984 server_base.cc:1034] running on GCE node
I20260924 11:34:20.667876 11984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:20.668853 11984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:20.670073 11984 hybrid_clock.cc:648] HybridClock initialized: now 1790249660670027 us; error 63 us; skew 500 ppm
I20260924 11:34:20.670400 11984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:34:20.672669 11984 webserver.cc:460] Webserver started at http://127.6.212.3:40297/ using document root <none> and password file <none>
I20260924 11:34:20.673357 11984 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:20.673485 11984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:20.673820 11984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:34:20.676980 11984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-2/data/instance:
uuid: "6d7642bf3f5d4e729298393aa431e3ee"
format_stamp: "Formatted at 2026-09-24 11:34:20 on dist-test-slave-cq7c"
server_key: "1a5150d24c3d279c137d321e2045a195"
server_key_iv: "0bbc7e92694f8cee91bef5c10c373abe"
server_key_version: "encryptionkey@0"
I20260924 11:34:20.677758 11984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance:
uuid: "6d7642bf3f5d4e729298393aa431e3ee"
format_stamp: "Formatted at 2026-09-24 11:34:20 on dist-test-slave-cq7c"
server_key: "1a5150d24c3d279c137d321e2045a195"
server_key_iv: "0bbc7e92694f8cee91bef5c10c373abe"
server_key_version: "encryptionkey@0"
I20260924 11:34:20.682866 11984 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 11:34:20.686547 11998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:20.687952 11984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 11:34:20.688125 11984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-2/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-2/wal
uuid: "6d7642bf3f5d4e729298393aa431e3ee"
format_stamp: "Formatted at 2026-09-24 11:34:20 on dist-test-slave-cq7c"
server_key: "1a5150d24c3d279c137d321e2045a195"
server_key_iv: "0bbc7e92694f8cee91bef5c10c373abe"
server_key_version: "encryptionkey@0"
I20260924 11:34:20.688355 11984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:20.693795 11984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:20.694592 11984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:34:20.694859 11984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:20.695842 11984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:34:20.697171 11984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:34:20.697297 11984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:20.697428 11984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:34:20.697515 11984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:20.731010 11984 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.3:43179
I20260924 11:34:20.731060 12110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.3:43179 every 8 connection(s)
I20260924 11:34:20.732339 11984 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
I20260924 11:34:20.739753 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 11984
I20260924 11:34:20.739935 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance
I20260924 11:34:20.740433 11852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:20.734857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:50093 (local address 127.6.212.62:35961)
0924 11:34:20.735028 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:20.735033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:20.736637 (+ 1604us) server_negotiation.cc:407] Connection header received
0924 11:34:20.738284 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:20.738290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:20.738363 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:20.738472 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:20.739721 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:20.739727 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:20.739732 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:34:20.739756 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:20.739785 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:34:20.739795 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:20.739934 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:20.740185 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 11:34:20.740233 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 11:34:20.740458 6992 external_mini_cluster.cc:1428] Setting key 307b7af866170db6395718340a6f8bbf
I20260924 11:34:20.741719 12113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:20.735429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35961 (local address 127.6.212.3:50093)
0924 11:34:20.736474 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:20.736514 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:20.738008 (+ 1494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:20.738701 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:20.738709 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:20.738795 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:20.739423 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:34:20.739426 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:20.739462 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:34:20.739545 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:20.739937 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:20.739951 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:20.739961 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 11:34:20.740264 (+ 303us) client_negotiation.cc:240] Negotiation successful
0924 11:34:20.740516 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":423,"threads_started":1}
I20260924 11:34:20.743017 12111 heartbeater.cc:347] Connected to a master server at 127.6.212.62:35961
I20260924 11:34:20.743310 12111 heartbeater.cc:464] Registering TS with master...
I20260924 11:34:20.743877 12111 heartbeater.cc:511] Master 127.6.212.62:35961 requested a full tablet report, sending...
I20260924 11:34:20.745144 11664 ts_manager.cc:194] Registered new tserver with Master: 6d7642bf3f5d4e729298393aa431e3ee (127.6.212.3:43179)
I20260924 11:34:20.755599 6992 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:34:20.762621 11852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:20.760099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60902 (local address 127.6.212.62:35961)
0924 11:34:20.760264 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:20.760269 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:20.760744 (+ 475us) server_negotiation.cc:407] Connection header received
0924 11:34:20.760943 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:20.760949 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:20.761011 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:20.761110 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:20.761989 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:20.761994 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:20.761999 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:34:20.762021 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:20.762049 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:34:20.762060 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:20.762183 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:20.762303 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 11:34:20.762361 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 11:34:20.766382 11664 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60902:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:34:20.773407 11664 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:34:20.793624 12124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:20.786307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50692 (local address 127.6.212.3:43179)
0924 11:34:20.788650 (+ 2343us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:20.788659 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:20.788693 (+ 34us) server_negotiation.cc:407] Connection header received
0924 11:34:20.788768 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:20.788778 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:20.789083 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:20.789261 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:20.792412 (+ 3151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:20.792423 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:20.792438 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:34:20.792465 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:20.792523 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:34:20.792545 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:20.792706 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:20.793153 (+ 447us) server_negotiation.cc:299] Negotiation successful
0924 11:34:20.793252 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2164,"thread_start_us":2007,"threads_started":1}
I20260924 11:34:20.794009 12125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:20.786848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36918 (local address 127.6.212.1:39603)
0924 11:34:20.788103 (+ 1255us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:20.788116 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:20.788849 (+ 733us) server_negotiation.cc:407] Connection header received
0924 11:34:20.788925 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:20.788932 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:20.789152 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:20.789321 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:20.792591 (+ 3270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:20.792599 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:20.792610 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:34:20.792646 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:20.792700 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:34:20.792722 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:20.792847 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:20.793404 (+ 557us) server_negotiation.cc:299] Negotiation successful
0924 11:34:20.793502 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1040,"thread_start_us":1062,"threads_started":1}
I20260924 11:34:20.794803 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:20.786277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:39603 (local address 127.0.0.1:36918)
0924 11:34:20.788557 (+ 2280us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:20.788595 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:20.788715 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:20.791667 (+ 2952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:20.791673 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:20.791695 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:20.792230 (+ 535us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:34:20.792236 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:20.792255 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:34:20.792292 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:20.792899 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:20.792912 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:20.792920 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 11:34:20.793104 (+ 184us) client_negotiation.cc:240] Negotiation successful
0924 11:34:20.793197 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2148,"thread_start_us":798,"threads_started":1}
I20260924 11:34:20.794826 12123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:20.786152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.3:43179 (local address 127.0.0.1:50692)
0924 11:34:20.786957 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:20.787016 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:20.787174 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:20.791045 (+ 3871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:20.791053 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:20.791177 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:20.792078 (+ 901us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:34:20.792083 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:20.792115 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:34:20.792230 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:20.792899 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:20.792912 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:20.792920 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 11:34:20.794027 (+ 1107us) client_negotiation.cc:240] Negotiation successful
0924 11:34:20.794173 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":2101,"threads_started":1}
I20260924 11:34:20.795037 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:20.788665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.2:39309 (local address 127.0.0.1:58908)
0924 11:34:20.791044 (+ 2379us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:20.791061 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:20.791145 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:20.792286 (+ 1141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:20.792290 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:20.792309 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:20.792944 (+ 635us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:34:20.792946 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:20.792957 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:34:20.792987 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:20.794222 (+ 1235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:20.794228 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:20.794233 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 11:34:20.794362 (+ 129us) client_negotiation.cc:240] Negotiation successful
0924 11:34:20.794411 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2266,"thread_start_us":2151,"threads_started":1}
I20260924 11:34:20.795787 12128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:20.789008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58908 (local address 127.6.212.2:39309)
0924 11:34:20.791703 (+ 2695us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:20.791712 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:20.791743 (+ 31us) server_negotiation.cc:407] Connection header received
0924 11:34:20.791814 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:20.791823 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:20.792102 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:20.792715 (+ 613us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:20.793982 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:20.793990 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:20.794003 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:34:20.794026 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:20.794076 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:34:20.794096 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:20.795286 (+ 1190us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:20.795389 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 11:34:20.795477 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2523,"thread_start_us":2394,"threads_started":1}
I20260924 11:34:20.798661 12046 tablet_service.cc:1467] Processing CreateTablet for tablet b4d9d31717934632813377af9a899649 (DEFAULT_TABLE table=test-table [id=0390e0e3f8714806a53d60f7c00a8036]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:34:20.798658 11785 tablet_service.cc:1467] Processing CreateTablet for tablet b4d9d31717934632813377af9a899649 (DEFAULT_TABLE table=test-table [id=0390e0e3f8714806a53d60f7c00a8036]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:34:20.798980 11916 tablet_service.cc:1467] Processing CreateTablet for tablet b4d9d31717934632813377af9a899649 (DEFAULT_TABLE table=test-table [id=0390e0e3f8714806a53d60f7c00a8036]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:34:20.799978 12046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4d9d31717934632813377af9a899649. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:34:20.800025 11785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4d9d31717934632813377af9a899649. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:34:20.800114 11916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4d9d31717934632813377af9a899649. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:34:20.811429 12129 tablet_bootstrap.cc:492] T b4d9d31717934632813377af9a899649 P 6d7642bf3f5d4e729298393aa431e3ee: Bootstrap starting.
I20260924 11:34:20.814496 12131 tablet_bootstrap.cc:492] T b4d9d31717934632813377af9a899649 P 26e38333eacd4d8a829d25c959e84f25: Bootstrap starting.
I20260924 11:34:20.815236 12129 tablet_bootstrap.cc:654] T b4d9d31717934632813377af9a899649 P 6d7642bf3f5d4e729298393aa431e3ee: Neither blocks nor log segments found. Creating new log.
I20260924 11:34:20.815716 12130 tablet_bootstrap.cc:492] T b4d9d31717934632813377af9a899649 P e1fcad3c1d23477dbdb666fcfbb81f78: Bootstrap starting.
I20260924 11:34:20.817883 12129 log.cc:824] T b4d9d31717934632813377af9a899649 P 6d7642bf3f5d4e729298393aa431e3ee: Log is configured to *not* fsync() on all Append() calls
I20260924 11:34:20.819181 12130 tablet_bootstrap.cc:654] T b4d9d31717934632813377af9a899649 P e1fcad3c1d23477dbdb666fcfbb81f78: Neither blocks nor log segments found. Creating new log.
I20260924 11:34:20.819319 12131 tablet_bootstrap.cc:654] T b4d9d31717934632813377af9a899649 P 26e38333eacd4d8a829d25c959e84f25: Neither blocks nor log segments found. Creating new log.
I20260924 11:34:20.820379 12131 log.cc:824] T b4d9d31717934632813377af9a899649 P 26e38333eacd4d8a829d25c959e84f25: Log is configured to *not* fsync() on all Append() calls
I20260924 11:34:20.820472 12130 log.cc:824] T b4d9d31717934632813377af9a899649 P e1fcad3c1d23477dbdb666fcfbb81f78: Log is configured to *not* fsync() on all Append() calls
I20260924 11:34:20.821427 12129 tablet_bootstrap.cc:492] T b4d9d31717934632813377af9a899649 P 6d7642bf3f5d4e729298393aa431e3ee: No bootstrap required, opened a new log
I20260924 11:34:20.821772 12129 ts_tablet_manager.cc:1397] T b4d9d31717934632813377af9a899649 P 6d7642bf3f5d4e729298393aa431e3ee: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.006s
I20260924 11:34:20.823531 12130 tablet_bootstrap.cc:492] T b4d9d31717934632813377af9a899649 P e1fcad3c1d23477dbdb666fcfbb81f78: No bootstrap required, opened a new log
I20260924 11:34:20.823534 12131 tablet_bootstrap.cc:492] T b4d9d31717934632813377af9a899649 P 26e38333eacd4d8a829d25c959e84f25: No bootstrap required, opened a new log
I20260924 11:34:20.823803 12130 ts_tablet_manager.cc:1397] T b4d9d31717934632813377af9a899649 P e1fcad3c1d23477dbdb666fcfbb81f78: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260924 11:34:20.823807 12131 ts_tablet_manager.cc:1397] T b4d9d31717934632813377af9a899649 P 26e38333eacd4d8a829d25c959e84f25: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260924 11:34:20.832242 12130 raft_consensus.cc:348] T b4d9d31717934632813377af9a899649 P e1fcad3c1d23477dbdb666fcfbb81f78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d7642bf3f5d4e729298393aa431e3ee" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43179 } } peers { permanent_uuid: "26e38333eacd4d8a829d25c959e84f25" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 39603 } } peers { permanent_uuid: "e1fcad3c1d23477dbdb666fcfbb81f78" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 39309 } }
I20260924 11:34:20.832690 12130 raft_consensus.cc:374] T b4d9d31717934632813377af9a899649 P e1fcad3c1d23477dbdb666fcfbb81f78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:34:20.832791 12130 raft_consensus.cc:729] T b4d9d31717934632813377af9a899649 P e1fcad3c1d23477dbdb666fcfbb81f78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1fcad3c1d23477dbdb666fcfbb81f78, State: Initialized, Role: FOLLOWER
I20260924 11:34:20.832962 12129 raft_consensus.cc:348] T b4d9d31717934632813377af9a899649 P 6d7642bf3f5d4e729298393aa431e3ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d7642bf3f5d4e729298393aa431e3ee" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43179 } } peers { permanent_uuid: "26e38333eacd4d8a829d25c959e84f25" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 39603 } } peers { permanent_uuid: "e1fcad3c1d23477dbdb666fcfbb81f78" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 39309 } }
I20260924 11:34:20.833400 12129 raft_consensus.cc:374] T b4d9d31717934632813377af9a899649 P 6d7642bf3f5d4e729298393aa431e3ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:34:20.833503 12129 raft_consensus.cc:729] T b4d9d31717934632813377af9a899649 P 6d7642bf3f5d4e729298393aa431e3ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d7642bf3f5d4e729298393aa431e3ee, State: Initialized, Role: FOLLOWER
I20260924 11:34:20.833626 12130 consensus_queue.cc:260] T b4d9d31717934632813377af9a899649 P e1fcad3c1d23477dbdb666fcfbb81f78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d7642bf3f5d4e729298393aa431e3ee" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43179 } } peers { permanent_uuid: "26e38333eacd4d8a829d25c959e84f25" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 39603 } } peers { permanent_uuid: "e1fcad3c1d23477dbdb666fcfbb81f78" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 39309 } }
I20260924 11:34:20.834084 12129 consensus_queue.cc:260] T b4d9d31717934632813377af9a899649 P 6d7642bf3f5d4e729298393aa431e3ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d7642bf3f5d4e729298393aa431e3ee" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43179 } } peers { permanent_uuid: "26e38333eacd4d8a829d25c959e84f25" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 39603 } } peers { permanent_uuid: "e1fcad3c1d23477dbdb666fcfbb81f78" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 39309 } }
I20260924 11:34:20.834399 12131 raft_consensus.cc:348] T b4d9d31717934632813377af9a899649 P 26e38333eacd4d8a829d25c959e84f25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d7642bf3f5d4e729298393aa431e3ee" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43179 } } peers { permanent_uuid: "26e38333eacd4d8a829d25c959e84f25" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 39603 } } peers { permanent_uuid: "e1fcad3c1d23477dbdb666fcfbb81f78" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 39309 } }
I20260924 11:34:20.834889 12131 raft_consensus.cc:374] T b4d9d31717934632813377af9a899649 P 26e38333eacd4d8a829d25c959e84f25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:34:20.835021 12131 raft_consensus.cc:729] T b4d9d31717934632813377af9a899649 P 26e38333eacd4d8a829d25c959e84f25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26e38333eacd4d8a829d25c959e84f25, State: Initialized, Role: FOLLOWER
I20260924 11:34:20.835122 11981 heartbeater.cc:503] Master 127.6.212.62:35961 was elected leader, sending a full tablet report...
I20260924 11:34:20.835853 12130 ts_tablet_manager.cc:1428] T b4d9d31717934632813377af9a899649 P e1fcad3c1d23477dbdb666fcfbb81f78: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260924 11:34:20.835888 12131 consensus_queue.cc:260] T b4d9d31717934632813377af9a899649 P 26e38333eacd4d8a829d25c959e84f25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d7642bf3f5d4e729298393aa431e3ee" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43179 } } peers { permanent_uuid: "26e38333eacd4d8a829d25c959e84f25" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 39603 } } peers { permanent_uuid: "e1fcad3c1d23477dbdb666fcfbb81f78" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 39309 } }
I20260924 11:34:20.840150 11850 heartbeater.cc:503] Master 127.6.212.62:35961 was elected leader, sending a full tablet report...
I20260924 11:34:20.840996 12131 ts_tablet_manager.cc:1428] T b4d9d31717934632813377af9a899649 P 26e38333eacd4d8a829d25c959e84f25: Time spent starting tablet: real 0.017s user 0.007s sys 0.009s
I20260924 11:34:20.842247 12129 ts_tablet_manager.cc:1428] T b4d9d31717934632813377af9a899649 P 6d7642bf3f5d4e729298393aa431e3ee: Time spent starting tablet: real 0.020s user 0.017s sys 0.002s
I20260924 11:34:20.842628 12111 heartbeater.cc:503] Master 127.6.212.62:35961 was elected leader, sending a full tablet report...
W20260924 11:34:20.864668 11851 tablet.cc:2369] T b4d9d31717934632813377af9a899649 P 26e38333eacd4d8a829d25c959e84f25: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:34:20.983989 12112 tablet.cc:2369] T b4d9d31717934632813377af9a899649 P 6d7642bf3f5d4e729298393aa431e3ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:34:21.029812 12135 raft_consensus.cc:482] T b4d9d31717934632813377af9a899649 P e1fcad3c1d23477dbdb666fcfbb81f78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:34:21.030299 12135 raft_consensus.cc:504] T b4d9d31717934632813377af9a899649 P e1fcad3c1d23477dbdb666fcfbb81f78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d7642bf3f5d4e729298393aa431e3ee" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43179 } } peers { permanent_uuid: "26e38333eacd4d8a829d25c959e84f25" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 39603 } } peers { permanent_uuid: "e1fcad3c1d23477dbdb666fcfbb81f78" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 39309 } }
I20260924 11:34:21.032502 12135 leader_election.cc:288] T b4d9d31717934632813377af9a899649 P e1fcad3c1d23477dbdb666fcfbb81f78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d7642bf3f5d4e729298393aa431e3ee (127.6.212.3:43179), 26e38333eacd4d8a829d25c959e84f25 (127.6.212.1:39603)
I20260924 11:34:21.036484 12124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:21.033034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:52881 (local address 127.6.212.3:43179)
0924 11:34:21.033286 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:21.033294 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:21.034157 (+ 863us) server_negotiation.cc:407] Connection header received
0924 11:34:21.034354 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:21.034357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:21.034403 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:21.034478 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:21.035741 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:21.035747 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:21.035751 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:34:21.035776 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:21.035806 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:34:21.035817 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:21.035969 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:21.036046 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 11:34:21.036112 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 11:34:21.036583 12125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:21.033145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:40101 (local address 127.6.212.1:39603)
0924 11:34:21.033369 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:21.033376 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:21.033966 (+ 590us) server_negotiation.cc:407] Connection header received
0924 11:34:21.034122 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:21.034127 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:21.034214 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:21.034338 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:21.035859 (+ 1521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:21.035869 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:21.035875 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:34:21.035900 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:21.035930 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:34:21.035941 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:21.036064 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:21.036191 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 11:34:21.036253 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260924 11:34:21.036811 12138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:21.032873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:39603 (local address 127.6.212.2:40101)
0924 11:34:21.033768 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:21.033804 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:21.033956 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:21.034649 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:21.034655 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:21.034673 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:21.035624 (+ 951us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:34:21.035626 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:21.035644 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:34:21.035687 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:21.036035 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:21.036039 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:21.036043 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 11:34:21.036179 (+ 136us) client_negotiation.cc:240] Negotiation successful
0924 11:34:21.036240 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"thread_start_us":426,"threads_started":1}
I20260924 11:34:21.037209 12139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:21.032979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.3:43179 (local address 127.6.212.2:52881)
0924 11:34:21.034011 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:21.034036 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:21.034160 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:21.034542 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:21.034546 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:21.034569 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:21.035433 (+ 864us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:34:21.035437 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:21.035463 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:34:21.035527 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:21.035940 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:21.035947 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:21.035952 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 11:34:21.036099 (+ 147us) client_negotiation.cc:240] Negotiation successful
0924 11:34:21.036147 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":869,"thread_start_us":574,"threads_started":1}
I20260924 11:34:21.037900 11805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d9d31717934632813377af9a899649" candidate_uuid: "e1fcad3c1d23477dbdb666fcfbb81f78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26e38333eacd4d8a829d25c959e84f25" is_pre_election: true
I20260924 11:34:21.038328 12066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d9d31717934632813377af9a899649" candidate_uuid: "e1fcad3c1d23477dbdb666fcfbb81f78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d7642bf3f5d4e729298393aa431e3ee" is_pre_election: true
I20260924 11:34:21.038465 11805 raft_consensus.cc:2454] T b4d9d31717934632813377af9a899649 P 26e38333eacd4d8a829d25c959e84f25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1fcad3c1d23477dbdb666fcfbb81f78 in term 0.
I20260924 11:34:21.038825 12066 raft_consensus.cc:2454] T b4d9d31717934632813377af9a899649 P 6d7642bf3f5d4e729298393aa431e3ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1fcad3c1d23477dbdb666fcfbb81f78 in term 0.
I20260924 11:34:21.039269 11871 leader_election.cc:302] T b4d9d31717934632813377af9a899649 P e1fcad3c1d23477dbdb666fcfbb81f78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26e38333eacd4d8a829d25c959e84f25, e1fcad3c1d23477dbdb666fcfbb81f78; no voters:
I20260924 11:34:21.039855 12135 raft_consensus.cc:2781] T b4d9d31717934632813377af9a899649 P e1fcad3c1d23477dbdb666fcfbb81f78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:34:21.040012 12135 raft_consensus.cc:482] T b4d9d31717934632813377af9a899649 P e1fcad3c1d23477dbdb666fcfbb81f78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:34:21.040119 12135 raft_consensus.cc:3037] T b4d9d31717934632813377af9a899649 P e1fcad3c1d23477dbdb666fcfbb81f78 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:34:21.043491 12135 raft_consensus.cc:504] T b4d9d31717934632813377af9a899649 P e1fcad3c1d23477dbdb666fcfbb81f78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d7642bf3f5d4e729298393aa431e3ee" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43179 } } peers { permanent_uuid: "26e38333eacd4d8a829d25c959e84f25" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 39603 } } peers { permanent_uuid: "e1fcad3c1d23477dbdb666fcfbb81f78" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 39309 } }
I20260924 11:34:21.044219 12135 leader_election.cc:288] T b4d9d31717934632813377af9a899649 P e1fcad3c1d23477dbdb666fcfbb81f78 [CANDIDATE]: Term 1 election: Requested vote from peers 6d7642bf3f5d4e729298393aa431e3ee (127.6.212.3:43179), 26e38333eacd4d8a829d25c959e84f25 (127.6.212.1:39603)
I20260924 11:34:21.044826 12066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d9d31717934632813377af9a899649" candidate_uuid: "e1fcad3c1d23477dbdb666fcfbb81f78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d7642bf3f5d4e729298393aa431e3ee"
I20260924 11:34:21.044890 11805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4d9d31717934632813377af9a899649" candidate_uuid: "e1fcad3c1d23477dbdb666fcfbb81f78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26e38333eacd4d8a829d25c959e84f25"
I20260924 11:34:21.045045 12066 raft_consensus.cc:3037] T b4d9d31717934632813377af9a899649 P 6d7642bf3f5d4e729298393aa431e3ee [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:34:21.045217 11805 raft_consensus.cc:3037] T b4d9d31717934632813377af9a899649 P 26e38333eacd4d8a829d25c959e84f25 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:34:21.047641 11805 raft_consensus.cc:2454] T b4d9d31717934632813377af9a899649 P 26e38333eacd4d8a829d25c959e84f25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1fcad3c1d23477dbdb666fcfbb81f78 in term 1.
I20260924 11:34:21.047644 12066 raft_consensus.cc:2454] T b4d9d31717934632813377af9a899649 P 6d7642bf3f5d4e729298393aa431e3ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1fcad3c1d23477dbdb666fcfbb81f78 in term 1.
I20260924 11:34:21.048199 11871 leader_election.cc:302] T b4d9d31717934632813377af9a899649 P e1fcad3c1d23477dbdb666fcfbb81f78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 26e38333eacd4d8a829d25c959e84f25, 6d7642bf3f5d4e729298393aa431e3ee, e1fcad3c1d23477dbdb666fcfbb81f78; no voters:
I20260924 11:34:21.048564 12135 raft_consensus.cc:2781] T b4d9d31717934632813377af9a899649 P e1fcad3c1d23477dbdb666fcfbb81f78 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:34:21.049078 12135 raft_consensus.cc:686] T b4d9d31717934632813377af9a899649 P e1fcad3c1d23477dbdb666fcfbb81f78 [term 1 LEADER]: Becoming Leader. State: Replica: e1fcad3c1d23477dbdb666fcfbb81f78, State: Running, Role: LEADER
I20260924 11:34:21.049604 12135 consensus_queue.cc:237] T b4d9d31717934632813377af9a899649 P e1fcad3c1d23477dbdb666fcfbb81f78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d7642bf3f5d4e729298393aa431e3ee" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43179 } } peers { permanent_uuid: "26e38333eacd4d8a829d25c959e84f25" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 39603 } } peers { permanent_uuid: "e1fcad3c1d23477dbdb666fcfbb81f78" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 39309 } }
I20260924 11:34:21.054929 11664 catalog_manager.cc:5495] T b4d9d31717934632813377af9a899649 P e1fcad3c1d23477dbdb666fcfbb81f78 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1fcad3c1d23477dbdb666fcfbb81f78 (127.6.212.2). New cstate: current_term: 1 leader_uuid: "e1fcad3c1d23477dbdb666fcfbb81f78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d7642bf3f5d4e729298393aa431e3ee" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26e38333eacd4d8a829d25c959e84f25" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 39603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1fcad3c1d23477dbdb666fcfbb81f78" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 39309 } health_report { overall_health: HEALTHY } } }
W20260924 11:34:21.057468 11982 tablet.cc:2369] T b4d9d31717934632813377af9a899649 P e1fcad3c1d23477dbdb666fcfbb81f78: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:34:21.090929 12128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:21.088732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58924 (local address 127.6.212.2:39309)
0924 11:34:21.088948 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:21.088953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:21.088973 (+ 20us) server_negotiation.cc:407] Connection header received
0924 11:34:21.089183 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:21.089187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:21.089273 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:21.089362 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:21.090246 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:21.090251 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:21.090254 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:34:21.090270 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:21.090304 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:34:21.090317 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:21.090425 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:21.090524 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 11:34:21.090574 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 11:34:21.099779 11805 raft_consensus.cc:1261] T b4d9d31717934632813377af9a899649 P 26e38333eacd4d8a829d25c959e84f25 [term 1 FOLLOWER]: Refusing update from remote peer e1fcad3c1d23477dbdb666fcfbb81f78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:34:21.099781 12066 raft_consensus.cc:1261] T b4d9d31717934632813377af9a899649 P 6d7642bf3f5d4e729298393aa431e3ee [term 1 FOLLOWER]: Refusing update from remote peer e1fcad3c1d23477dbdb666fcfbb81f78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:34:21.100819 12140 consensus_queue.cc:1035] T b4d9d31717934632813377af9a899649 P e1fcad3c1d23477dbdb666fcfbb81f78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26e38333eacd4d8a829d25c959e84f25" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 39603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:34:21.101177 12135 consensus_queue.cc:1035] T b4d9d31717934632813377af9a899649 P e1fcad3c1d23477dbdb666fcfbb81f78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d7642bf3f5d4e729298393aa431e3ee" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:34:21.112612 12143 mvcc.cc:205] Tried to move back new op lower bound from 7332862611854172160 to 7332862611663368192. Current Snapshot: MvccSnapshot[applied={T|T < 7332862611854172160}]
I20260924 11:34:21.131762 11663 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:34:21.132081 11663 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:34:21.135673 11663 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 09bcc0e367ee44a9bcaf3fb6d8d1d1c6: Sending DeleteTablet for 3 replicas of tablet b4d9d31717934632813377af9a899649
I20260924 11:34:21.136895 12046 tablet_service.cc:1514] Processing DeleteTablet for tablet b4d9d31717934632813377af9a899649 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:34:21 UTC) from {username='slave'} at 127.0.0.1:50692
I20260924 11:34:21.137028 11785 tablet_service.cc:1514] Processing DeleteTablet for tablet b4d9d31717934632813377af9a899649 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:34:21 UTC) from {username='slave'} at 127.0.0.1:36918
I20260924 11:34:21.137076 11916 tablet_service.cc:1514] Processing DeleteTablet for tablet b4d9d31717934632813377af9a899649 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:34:21 UTC) from {username='slave'} at 127.0.0.1:58908
I20260924 11:34:21.142881 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 11723
I20260924 11:34:21.153910 12156 tablet_replica.cc:330] T b4d9d31717934632813377af9a899649 P e1fcad3c1d23477dbdb666fcfbb81f78: stopping tablet replica
I20260924 11:34:21.154593 12156 raft_consensus.cc:2229] T b4d9d31717934632813377af9a899649 P e1fcad3c1d23477dbdb666fcfbb81f78 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:34:21.155536 12156 raft_consensus.cc:2258] T b4d9d31717934632813377af9a899649 P e1fcad3c1d23477dbdb666fcfbb81f78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:34:21.156738 12155 tablet_replica.cc:330] T b4d9d31717934632813377af9a899649 P 26e38333eacd4d8a829d25c959e84f25: stopping tablet replica
I20260924 11:34:21.157298 12155 raft_consensus.cc:2229] T b4d9d31717934632813377af9a899649 P 26e38333eacd4d8a829d25c959e84f25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:34:21.157950 12155 raft_consensus.cc:2258] T b4d9d31717934632813377af9a899649 P 26e38333eacd4d8a829d25c959e84f25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:34:21.160405 12154 tablet_replica.cc:330] T b4d9d31717934632813377af9a899649 P 6d7642bf3f5d4e729298393aa431e3ee: stopping tablet replica
I20260924 11:34:21.160799 12154 raft_consensus.cc:2229] T b4d9d31717934632813377af9a899649 P 6d7642bf3f5d4e729298393aa431e3ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:34:21.161406 12156 ts_tablet_manager.cc:1905] T b4d9d31717934632813377af9a899649 P e1fcad3c1d23477dbdb666fcfbb81f78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:34:21.162168 12125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:21.144262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36924 (local address 127.6.212.1:39603)
0924 11:34:21.144440 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:21.144446 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:21.158606 (+ 14160us) server_negotiation.cc:407] Connection header received
0924 11:34:21.159161 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:21.159168 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:21.159267 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:21.159347 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:21.161323 (+ 1976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:21.161329 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:21.161335 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:34:21.161356 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:21.161392 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:34:21.161407 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:21.161505 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:21.161798 (+ 293us) server_negotiation.cc:299] Negotiation successful
0924 11:34:21.161864 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 11:34:21.162424 12155 ts_tablet_manager.cc:1905] T b4d9d31717934632813377af9a899649 P 26e38333eacd4d8a829d25c959e84f25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:34:21.163396 11845 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:34:21.163661 12154 raft_consensus.cc:2258] T b4d9d31717934632813377af9a899649 P 6d7642bf3f5d4e729298393aa431e3ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:34:21.165735 12154 ts_tablet_manager.cc:1905] T b4d9d31717934632813377af9a899649 P 6d7642bf3f5d4e729298393aa431e3ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:34:21.172634 12154 ts_tablet_manager.cc:1918] T b4d9d31717934632813377af9a899649 P 6d7642bf3f5d4e729298393aa431e3ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:34:21.173120 12156 ts_tablet_manager.cc:1918] T b4d9d31717934632813377af9a899649 P e1fcad3c1d23477dbdb666fcfbb81f78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:34:21.173381 12154 log.cc:1196] T b4d9d31717934632813377af9a899649 P 6d7642bf3f5d4e729298393aa431e3ee: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-2/wal/wals/b4d9d31717934632813377af9a899649
I20260924 11:34:21.173375 12156 log.cc:1196] T b4d9d31717934632813377af9a899649 P e1fcad3c1d23477dbdb666fcfbb81f78: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790249617057452-6992-0/minicluster-data/ts-1/wal/wals/b4d9d31717934632813377af9a899649
I20260924 11:34:21.174157 12156 ts_tablet_manager.cc:1939] T b4d9d31717934632813377af9a899649 P e1fcad3c1d23477dbdb666fcfbb81f78: Deleting consensus metadata
I20260924 11:34:21.175658 12154 ts_tablet_manager.cc:1939] T b4d9d31717934632813377af9a899649 P 6d7642bf3f5d4e729298393aa431e3ee: Deleting consensus metadata
I20260924 11:34:21.176512 11648 catalog_manager.cc:4841] TS e1fcad3c1d23477dbdb666fcfbb81f78 (127.6.212.2:39309): tablet b4d9d31717934632813377af9a899649 (table test-table [id=0390e0e3f8714806a53d60f7c00a8036]) successfully deleted
I20260924 11:34:21.177449 11648 catalog_manager.cc:4841] TS 6d7642bf3f5d4e729298393aa431e3ee (127.6.212.3:43179): tablet b4d9d31717934632813377af9a899649 (table test-table [id=0390e0e3f8714806a53d60f7c00a8036]) successfully deleted
I20260924 11:34:21.272679 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 11723
I20260924 11:34:21.291730 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 11854
W20260924 11:34:21.291749 11650 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.212.1:39603 (error 108)
I20260924 11:34:21.293710 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:21.293160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:39603 (local address 127.0.0.1:36928)
0924 11:34:21.293362 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:21.293508 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.1:39603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260924 11:34:21.293963 11650 catalog_manager.cc:4584] TS 26e38333eacd4d8a829d25c959e84f25 (127.6.212.1:39603): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b4d9d31717934632813377af9a899649: Network error: Client connection negotiation failed: client connection to 127.6.212.1:39603: connect: Connection refused (error 111)
I20260924 11:34:21.295327 12128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:21.292944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58938 (local address 127.6.212.2:39309)
0924 11:34:21.293135 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:21.293140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:21.293160 (+ 20us) server_negotiation.cc:407] Connection header received
0924 11:34:21.293270 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:21.293276 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:21.293343 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:21.293468 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:21.294544 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:21.294551 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:21.294556 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:34:21.294575 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:21.294618 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:34:21.294632 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:21.294796 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:21.294912 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 11:34:21.294972 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 11:34:21.296129 11976 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:34:21.347759 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:21.347199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:39603 (local address 127.0.0.1:36942)
0924 11:34:21.347454 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:21.347591 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.1:39603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260924 11:34:21.348930 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:21.348532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:39603 (local address 127.0.0.1:36946)
0924 11:34:21.348701 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:21.348786 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.1:39603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 11:34:21.409071 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 11854
I20260924 11:34:21.415735 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:21.408936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:39603 (local address 127.0.0.1:36962)
0924 11:34:21.415449 (+ 6513us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:21.415583 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.1:39603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6334}
I20260924 11:34:21.416777 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:21.416435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:39603 (local address 127.0.0.1:36978)
0924 11:34:21.416577 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:21.416668 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.1:39603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 11:34:21.428102 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 11984
I20260924 11:34:21.430964 12124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:21.428907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50698 (local address 127.6.212.3:43179)
0924 11:34:21.429184 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:21.429189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:21.429208 (+ 19us) server_negotiation.cc:407] Connection header received
0924 11:34:21.429265 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:21.429269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:21.429353 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:21.429446 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:21.430333 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:21.430338 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:21.430341 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:34:21.430356 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:21.430386 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:34:21.430399 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:21.430516 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:21.430551 (+ 35us) server_negotiation.cc:299] Negotiation successful
0924 11:34:21.430657 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260924 11:34:21.431684 12106 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:34:21.497231 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:21.496718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:39603 (local address 127.0.0.1:36994)
0924 11:34:21.496963 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:21.497047 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.1:39603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260924 11:34:21.498221 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:21.497855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:39603 (local address 127.0.0.1:37008)
0924 11:34:21.498046 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:21.498107 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.1:39603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260924 11:34:21.521032 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 11984
I20260924 11:34:21.537010 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 11633
I20260924 11:34:21.538262 11694 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:34:21.642580 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 11633
2026-09-24T11:34:21Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2788 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 11:34:21.684528 6992 mini_oidc.cc:152] Starting JWKS server
I20260924 11:34:21.687060 6992 webserver.cc:460] Webserver started at https://127.0.0.1:45969/ using document root <none> and password file <none>
I20260924 11:34:21.687335 6992 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 11:34:21.687999 6992 webserver.cc:460] Webserver started at http://127.0.0.1:42421/ using document root <none> and password file <none>
2026-09-24T11:34:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:34:21Z Disabled control of system clock
I20260924 11:34:21.718413 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:40359
--webserver_interface=127.6.212.62
--webserver_port=0
--builtin_ntp_servers=127.6.212.20:40447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:40359
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45969/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790249617057452-6992-0/testchainca.cert with env {}
W20260924 11:34:22.050850 12168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:22.051210 12168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:22.051412 12168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:22.059377 12168 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:34:22.059496 12168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:22.059572 12168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:22.059662 12168 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:34:22.059741 12168 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 11:34:22.059840 12168 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 11:34:22.059907 12168 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45969/jwks.json
I20260924 11:34:22.069953 12168 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:40447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:40359
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:40359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45969/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.212.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790249617057452-6992-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:34:22.071329 12168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:22.072741 12168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:22.084553 12173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:22.084601 12174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:22.084635 12176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:22.086333 12168 server_base.cc:1034] running on GCE node
I20260924 11:34:22.087196 12168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:22.088734 12168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:22.089962 12168 hybrid_clock.cc:648] HybridClock initialized: now 1790249662089908 us; error 70 us; skew 500 ppm
I20260924 11:34:22.090354 12168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:34:22.092756 12168 webserver.cc:460] Webserver started at http://127.6.212.62:40363/ using document root <none> and password file <none>
I20260924 11:34:22.093458 12168 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:22.093595 12168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:22.093899 12168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:34:22.096777 12168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "6dc30e0cffd24e57a7c78e2a7b808baa"
format_stamp: "Formatted at 2026-09-24 11:34:22 on dist-test-slave-cq7c"
server_key: "e4cca457b25d3dbd2ce30d6f316aeb33"
server_key_iv: "6b507dfe2f5f0e7c616ae79382edeb54"
server_key_version: "encryptionkey@0"
I20260924 11:34:22.097551 12168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "6dc30e0cffd24e57a7c78e2a7b808baa"
format_stamp: "Formatted at 2026-09-24 11:34:22 on dist-test-slave-cq7c"
server_key: "e4cca457b25d3dbd2ce30d6f316aeb33"
server_key_iv: "6b507dfe2f5f0e7c616ae79382edeb54"
server_key_version: "encryptionkey@0"
I20260924 11:34:22.102659 12168 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260924 11:34:22.106178 12182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:22.107527 12168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:34:22.107712 12168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "6dc30e0cffd24e57a7c78e2a7b808baa"
format_stamp: "Formatted at 2026-09-24 11:34:22 on dist-test-slave-cq7c"
server_key: "e4cca457b25d3dbd2ce30d6f316aeb33"
server_key_iv: "6b507dfe2f5f0e7c616ae79382edeb54"
server_key_version: "encryptionkey@0"
I20260924 11:34:22.107921 12168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:22.121965 12168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:22.122752 12168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:34:22.123050 12168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:22.141633 12168 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.62:40359
I20260924 11:34:22.141656 12233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.62:40359 every 8 connection(s)
I20260924 11:34:22.143033 12168 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:34:22.146867 12234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:34:22.147040 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 12168
I20260924 11:34:22.147230 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:34:22.147902 6992 external_mini_cluster.cc:1428] Setting key cee68e7d9877179706c927451b40c119
I20260924 11:34:22.160375 12234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dc30e0cffd24e57a7c78e2a7b808baa: Bootstrap starting.
I20260924 11:34:22.161857 12237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:22.149690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56748 (local address 127.6.212.62:40359)
0924 11:34:22.150715 (+ 1025us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:22.150726 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:22.150770 (+ 44us) server_negotiation.cc:407] Connection header received
0924 11:34:22.152768 (+ 1998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:22.152789 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:22.153262 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:22.153609 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:22.154351 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:22.155679 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:22.157588 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:22.158253 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:22.158379 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:22.159600 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:22.159607 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:22.159627 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:34:22.160141 (+ 514us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:22.160212 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:34:22.160232 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:22.160314 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:22.160593 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 11:34:22.160922 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":389,"threads_started":1}
I20260924 11:34:22.163622 12234 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6dc30e0cffd24e57a7c78e2a7b808baa: Neither blocks nor log segments found. Creating new log.
I20260924 11:34:22.164736 12234 log.cc:824] T 00000000000000000000000000000000 P 6dc30e0cffd24e57a7c78e2a7b808baa: Log is configured to *not* fsync() on all Append() calls
I20260924 11:34:22.165627 6992 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:34:22.167991 12234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dc30e0cffd24e57a7c78e2a7b808baa: No bootstrap required, opened a new log
I20260924 11:34:22.176126 12234 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6dc30e0cffd24e57a7c78e2a7b808baa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dc30e0cffd24e57a7c78e2a7b808baa" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 40359 } }
I20260924 11:34:22.176446 12234 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6dc30e0cffd24e57a7c78e2a7b808baa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:34:22.176525 12234 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6dc30e0cffd24e57a7c78e2a7b808baa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dc30e0cffd24e57a7c78e2a7b808baa, State: Initialized, Role: FOLLOWER
I20260924 11:34:22.177088 12234 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6dc30e0cffd24e57a7c78e2a7b808baa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dc30e0cffd24e57a7c78e2a7b808baa" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 40359 } }
I20260924 11:34:22.177277 12234 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6dc30e0cffd24e57a7c78e2a7b808baa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:34:22.177371 12234 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6dc30e0cffd24e57a7c78e2a7b808baa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:34:22.177491 12234 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6dc30e0cffd24e57a7c78e2a7b808baa [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:34:22.179796 12234 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6dc30e0cffd24e57a7c78e2a7b808baa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dc30e0cffd24e57a7c78e2a7b808baa" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 40359 } }
I20260924 11:34:22.180331 12234 leader_election.cc:302] T 00000000000000000000000000000000 P 6dc30e0cffd24e57a7c78e2a7b808baa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6dc30e0cffd24e57a7c78e2a7b808baa; no voters:
I20260924 11:34:22.180864 12234 leader_election.cc:288] T 00000000000000000000000000000000 P 6dc30e0cffd24e57a7c78e2a7b808baa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:34:22.181018 12239 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6dc30e0cffd24e57a7c78e2a7b808baa [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:34:22.181735 12239 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6dc30e0cffd24e57a7c78e2a7b808baa [term 1 LEADER]: Becoming Leader. State: Replica: 6dc30e0cffd24e57a7c78e2a7b808baa, State: Running, Role: LEADER
I20260924 11:34:22.182264 12239 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6dc30e0cffd24e57a7c78e2a7b808baa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dc30e0cffd24e57a7c78e2a7b808baa" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 40359 } }
I20260924 11:34:22.182932 12234 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6dc30e0cffd24e57a7c78e2a7b808baa [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:34:22.185369 12241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dc30e0cffd24e57a7c78e2a7b808baa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6dc30e0cffd24e57a7c78e2a7b808baa. Latest consensus state: current_term: 1 leader_uuid: "6dc30e0cffd24e57a7c78e2a7b808baa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dc30e0cffd24e57a7c78e2a7b808baa" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 40359 } } }
I20260924 11:34:22.185556 12240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dc30e0cffd24e57a7c78e2a7b808baa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6dc30e0cffd24e57a7c78e2a7b808baa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6dc30e0cffd24e57a7c78e2a7b808baa" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 40359 } } }
I20260924 11:34:22.185875 12241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dc30e0cffd24e57a7c78e2a7b808baa [sys.catalog]: This master's current role is: LEADER
I20260924 11:34:22.185937 12240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dc30e0cffd24e57a7c78e2a7b808baa [sys.catalog]: This master's current role is: LEADER
I20260924 11:34:22.187397 12245 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:34:22.192529 12245 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:34:22.202601 12245 catalog_manager.cc:1347] Generated new cluster ID: 69d122c1d8e14a599246c5186f26e83a
I20260924 11:34:22.202749 12245 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:34:22.206262 12256 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 11:34:22.223708 12245 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:34:22.225008 12245 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:34:22.230221 12245 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6dc30e0cffd24e57a7c78e2a7b808baa: Generated new TSK 0
I20260924 11:34:22.231309 12245 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 11:34:22.270726 12264 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 11:34:22.270985 12264 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 11:34:22.272192 12237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:22.249175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56754 (local address 127.6.212.62:40359)
0924 11:34:22.249376 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:22.249382 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:22.249859 (+ 477us) server_negotiation.cc:407] Connection header received
0924 11:34:22.250155 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:22.250161 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:22.250256 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:22.250416 (+ 160us) server_negotiation.cc:226] Negotiated authn=JWT
0924 11:34:22.250922 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:22.252114 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:22.253742 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:22.254406 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:22.254544 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:22.255108 (+ 564us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 11:34:22.271409 (+ 16301us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45969/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 11:34:22.271850 (+ 441us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56754: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45969/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":62}
W20260924 11:34:22.272329 12237 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56754: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45969/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 11:34:22.273787 12263 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:34:22.249055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:40359 (local address 127.0.0.1:56754)
0924 11:34:22.249751 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:22.249771 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:22.249922 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:22.250442 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:22.250446 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:22.250456 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0924 11:34:22.250759 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:22.250767 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:22.252302 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:22.252307 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:22.253590 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:22.253604 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:22.254622 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:22.254627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:22.254764 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:22.254828 (+ 64us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 11:34:22.271767 (+ 16939us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45969/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 11:34:22.272156 (+ 389us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.212.62:40359: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45969/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":354,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.6.212.62:40359: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45969/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 11:34:22.275933 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 12168
I20260924 11:34:22.277140 12229 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:34:22.369313 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 12168
2026-09-24T11:34:22Z chronyd exiting
I20260924 11:34:22.419866 6992 test_util.cc:175] -----------------------------------------------
I20260924 11:34:22.420011 6992 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790249617057452-6992-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (739 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 11:34:22.423663 6992 mini_oidc.cc:152] Starting JWKS server
I20260924 11:34:22.425542 6992 webserver.cc:460] Webserver started at https://127.0.0.1:35561/ using document root <none> and password file <none>
I20260924 11:34:22.425786 6992 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 11:34:22.426380 6992 webserver.cc:460] Webserver started at http://127.0.0.1:42221/ using document root <none> and password file <none>
2026-09-24T11:34:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:34:22Z Disabled control of system clock
I20260924 11:34:22.454111 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:44133
--webserver_interface=127.6.212.62
--webserver_port=0
--builtin_ntp_servers=127.6.212.20:38593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:44133
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35561/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790249617057452-6992-0/testchainca.cert with env {}
W20260924 11:34:22.773924 12271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:22.774274 12271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:22.774441 12271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:22.781675 12271 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:34:22.781859 12271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:22.781953 12271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:22.782065 12271 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:34:22.782153 12271 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 11:34:22.782243 12271 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 11:34:22.782315 12271 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35561/jwks.json
I20260924 11:34:22.792403 12271 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:38593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:44133
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:44133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35561/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.212.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790249617057452-6992-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:34:22.793828 12271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:22.795176 12271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:22.807278 12277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:22.807215 12279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:22.807327 12276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:22.808029 12271 server_base.cc:1034] running on GCE node
I20260924 11:34:22.808830 12271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:22.810267 12271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:22.811483 12271 hybrid_clock.cc:648] HybridClock initialized: now 1790249662811394 us; error 73 us; skew 500 ppm
I20260924 11:34:22.811852 12271 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:34:22.814258 12271 webserver.cc:460] Webserver started at http://127.6.212.62:42189/ using document root <none> and password file <none>
I20260924 11:34:22.814951 12271 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:22.815078 12271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:22.815474 12271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:34:22.818271 12271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "556b1ca843d8450a825b04efa3abeddd"
format_stamp: "Formatted at 2026-09-24 11:34:22 on dist-test-slave-cq7c"
server_key: "54791d1a84522b3e714f95c341e09b94"
server_key_iv: "2b51086bd51dfb27889583b106d3330f"
server_key_version: "encryptionkey@0"
I20260924 11:34:22.819048 12271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "556b1ca843d8450a825b04efa3abeddd"
format_stamp: "Formatted at 2026-09-24 11:34:22 on dist-test-slave-cq7c"
server_key: "54791d1a84522b3e714f95c341e09b94"
server_key_iv: "2b51086bd51dfb27889583b106d3330f"
server_key_version: "encryptionkey@0"
I20260924 11:34:22.824313 12271 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260924 11:34:22.827977 12285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:22.829406 12271 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260924 11:34:22.829571 12271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "556b1ca843d8450a825b04efa3abeddd"
format_stamp: "Formatted at 2026-09-24 11:34:22 on dist-test-slave-cq7c"
server_key: "54791d1a84522b3e714f95c341e09b94"
server_key_iv: "2b51086bd51dfb27889583b106d3330f"
server_key_version: "encryptionkey@0"
I20260924 11:34:22.829798 12271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:22.851442 12271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:22.852267 12271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:34:22.852573 12271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:22.871135 12271 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.62:44133
I20260924 11:34:22.871143 12336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.62:44133 every 8 connection(s)
I20260924 11:34:22.872545 12271 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:34:22.874185 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 12271
I20260924 11:34:22.874382 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:34:22.875221 6992 external_mini_cluster.cc:1428] Setting key 7e533730ae7801145b65bfe96bcab1be
I20260924 11:34:22.877381 12337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:34:22.888378 12340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:22.877153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45334 (local address 127.6.212.62:44133)
0924 11:34:22.878165 (+ 1012us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:22.878177 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:22.878217 (+ 40us) server_negotiation.cc:407] Connection header received
0924 11:34:22.879823 (+ 1606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:22.879844 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:22.880279 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:22.880704 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:22.881283 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:22.882602 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:22.883917 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:22.884577 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:22.884783 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:22.886036 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:22.886044 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:22.886058 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:34:22.886593 (+ 535us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:22.886690 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:34:22.886712 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:22.886848 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:22.887076 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 11:34:22.887448 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":373,"threads_started":1}
I20260924 11:34:22.889604 12337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 556b1ca843d8450a825b04efa3abeddd: Bootstrap starting.
I20260924 11:34:22.892112 6992 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:34:22.893108 12337 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 556b1ca843d8450a825b04efa3abeddd: Neither blocks nor log segments found. Creating new log.
I20260924 11:34:22.894266 12337 log.cc:824] T 00000000000000000000000000000000 P 556b1ca843d8450a825b04efa3abeddd: Log is configured to *not* fsync() on all Append() calls
I20260924 11:34:22.897249 12337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 556b1ca843d8450a825b04efa3abeddd: No bootstrap required, opened a new log
I20260924 11:34:22.905263 12337 raft_consensus.cc:348] T 00000000000000000000000000000000 P 556b1ca843d8450a825b04efa3abeddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "556b1ca843d8450a825b04efa3abeddd" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 44133 } }
I20260924 11:34:22.905656 12337 raft_consensus.cc:374] T 00000000000000000000000000000000 P 556b1ca843d8450a825b04efa3abeddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:34:22.905779 12337 raft_consensus.cc:729] T 00000000000000000000000000000000 P 556b1ca843d8450a825b04efa3abeddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 556b1ca843d8450a825b04efa3abeddd, State: Initialized, Role: FOLLOWER
I20260924 11:34:22.906342 12337 consensus_queue.cc:260] T 00000000000000000000000000000000 P 556b1ca843d8450a825b04efa3abeddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "556b1ca843d8450a825b04efa3abeddd" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 44133 } }
I20260924 11:34:22.906598 12337 raft_consensus.cc:388] T 00000000000000000000000000000000 P 556b1ca843d8450a825b04efa3abeddd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:34:22.906756 12337 raft_consensus.cc:482] T 00000000000000000000000000000000 P 556b1ca843d8450a825b04efa3abeddd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:34:22.906955 12337 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 556b1ca843d8450a825b04efa3abeddd [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:34:22.909384 12337 raft_consensus.cc:504] T 00000000000000000000000000000000 P 556b1ca843d8450a825b04efa3abeddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "556b1ca843d8450a825b04efa3abeddd" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 44133 } }
I20260924 11:34:22.909914 12337 leader_election.cc:302] T 00000000000000000000000000000000 P 556b1ca843d8450a825b04efa3abeddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 556b1ca843d8450a825b04efa3abeddd; no voters:
I20260924 11:34:22.910483 12337 leader_election.cc:288] T 00000000000000000000000000000000 P 556b1ca843d8450a825b04efa3abeddd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:34:22.910584 12342 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 556b1ca843d8450a825b04efa3abeddd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:34:22.911319 12342 raft_consensus.cc:686] T 00000000000000000000000000000000 P 556b1ca843d8450a825b04efa3abeddd [term 1 LEADER]: Becoming Leader. State: Replica: 556b1ca843d8450a825b04efa3abeddd, State: Running, Role: LEADER
I20260924 11:34:22.911836 12342 consensus_queue.cc:237] T 00000000000000000000000000000000 P 556b1ca843d8450a825b04efa3abeddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "556b1ca843d8450a825b04efa3abeddd" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 44133 } }
I20260924 11:34:22.912592 12337 sys_catalog.cc:564] T 00000000000000000000000000000000 P 556b1ca843d8450a825b04efa3abeddd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:34:22.914794 12343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 556b1ca843d8450a825b04efa3abeddd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "556b1ca843d8450a825b04efa3abeddd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "556b1ca843d8450a825b04efa3abeddd" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 44133 } } }
I20260924 11:34:22.914831 12344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 556b1ca843d8450a825b04efa3abeddd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 556b1ca843d8450a825b04efa3abeddd. Latest consensus state: current_term: 1 leader_uuid: "556b1ca843d8450a825b04efa3abeddd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "556b1ca843d8450a825b04efa3abeddd" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 44133 } } }
I20260924 11:34:22.915254 12344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 556b1ca843d8450a825b04efa3abeddd [sys.catalog]: This master's current role is: LEADER
I20260924 11:34:22.915254 12343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 556b1ca843d8450a825b04efa3abeddd [sys.catalog]: This master's current role is: LEADER
I20260924 11:34:22.919988 12349 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:34:22.925065 12349 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:34:22.934178 12349 catalog_manager.cc:1347] Generated new cluster ID: efad6fbcf32a438bbeb99c9a388d6012
I20260924 11:34:22.934283 12349 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:34:22.953112 12349 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:34:22.954478 12349 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:34:22.967552 12349 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 556b1ca843d8450a825b04efa3abeddd: Generated new TSK 0
I20260924 11:34:22.968397 12349 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 11:34:23.013530 12367 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 11:34:23.013661 12367 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 11:34:23.015048 12340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:22.987789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45344 (local address 127.6.212.62:44133)
0924 11:34:22.988080 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:22.988089 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:22.988373 (+ 284us) server_negotiation.cc:407] Connection header received
0924 11:34:22.988649 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:22.988652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:22.988724 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:22.988870 (+ 146us) server_negotiation.cc:226] Negotiated authn=JWT
0924 11:34:22.989324 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:22.990059 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:22.991416 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:22.991878 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:22.992033 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:22.992523 (+ 490us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 11:34:23.014257 (+ 21734us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35561/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 11:34:23.014673 (+ 416us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45344: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35561/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":59}
W20260924 11:34:23.015192 12340 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45344: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35561/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260924 11:34:23.015213 12366 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:34:22.987640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:44133 (local address 127.0.0.1:45344)
0924 11:34:22.988274 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:22.988289 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:22.988448 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:22.988888 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:22.988893 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:22.988906 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0924 11:34:22.989180 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:22.989188 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:22.990193 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:22.990198 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:22.991247 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:22.991257 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:22.992050 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:22.992056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:22.992225 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:22.992256 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 11:34:23.014509 (+ 22253us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35561/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 11:34:23.014809 (+ 300us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.212.62:44133: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35561/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":359,"threads_started":1}
I20260924 11:34:23.016261 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 12271
I20260924 11:34:23.017617 12332 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:34:23.111127 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 12271
2026-09-24T11:34:23Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (742 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 11:34:23.166477 6992 mini_oidc.cc:152] Starting JWKS server
I20260924 11:34:23.168437 6992 webserver.cc:460] Webserver started at https://127.0.0.1:46275/ using document root <none> and password file <none>
I20260924 11:34:23.168681 6992 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 11:34:23.169309 6992 webserver.cc:460] Webserver started at http://127.0.0.1:45059/ using document root <none> and password file <none>
2026-09-24T11:34:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:34:23Z Disabled control of system clock
I20260924 11:34:23.196521 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:36247
--webserver_interface=127.6.212.62
--webserver_port=0
--builtin_ntp_servers=127.6.212.20:43755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:36247
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46275/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 11:34:23.518338 12374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:23.518612 12374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:23.518695 12374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:23.525977 12374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:34:23.526083 12374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:23.526121 12374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:23.526177 12374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:34:23.526221 12374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 11:34:23.526280 12374 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 11:34:23.526314 12374 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46275/jwks.json
I20260924 11:34:23.535490 12374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:43755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:36247
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:36247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46275/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.212.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:34:23.536721 12374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:23.538513 12374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:23.550328 12379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:23.550288 12382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:23.550242 12380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:23.551007 12374 server_base.cc:1034] running on GCE node
I20260924 11:34:23.551834 12374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:23.553360 12374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:23.554561 12374 hybrid_clock.cc:648] HybridClock initialized: now 1790249663554503 us; error 57 us; skew 500 ppm
I20260924 11:34:23.554901 12374 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:34:23.557286 12374 webserver.cc:460] Webserver started at http://127.6.212.62:33103/ using document root <none> and password file <none>
I20260924 11:34:23.557982 12374 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:23.558096 12374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:23.558425 12374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:34:23.561291 12374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "badb2f44bbdc426dafa858e16d238c49"
format_stamp: "Formatted at 2026-09-24 11:34:23 on dist-test-slave-cq7c"
server_key: "cdc88913bb3f12d2a0499f8682210f91"
server_key_iv: "381f14527f9f6d1ff5a245309be1a31c"
server_key_version: "encryptionkey@0"
I20260924 11:34:23.562130 12374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "badb2f44bbdc426dafa858e16d238c49"
format_stamp: "Formatted at 2026-09-24 11:34:23 on dist-test-slave-cq7c"
server_key: "cdc88913bb3f12d2a0499f8682210f91"
server_key_iv: "381f14527f9f6d1ff5a245309be1a31c"
server_key_version: "encryptionkey@0"
I20260924 11:34:23.567186 12374 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 11:34:23.570618 12388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:23.572108 12374 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:34:23.572294 12374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "badb2f44bbdc426dafa858e16d238c49"
format_stamp: "Formatted at 2026-09-24 11:34:23 on dist-test-slave-cq7c"
server_key: "cdc88913bb3f12d2a0499f8682210f91"
server_key_iv: "381f14527f9f6d1ff5a245309be1a31c"
server_key_version: "encryptionkey@0"
I20260924 11:34:23.572505 12374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:23.592271 12374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:23.593094 12374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:34:23.593395 12374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:23.611479 12439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.62:36247 every 8 connection(s)
I20260924 11:34:23.611500 12374 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.62:36247
I20260924 11:34:23.612823 12374 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:34:23.614650 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 12374
I20260924 11:34:23.614845 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:34:23.615515 6992 external_mini_cluster.cc:1428] Setting key e7e2a339911538f88a63b5aca80b25bb
I20260924 11:34:23.616715 12440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:34:23.628015 12443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:23.617247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42090 (local address 127.6.212.62:36247)
0924 11:34:23.618155 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:23.618166 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:23.618222 (+ 56us) server_negotiation.cc:407] Connection header received
0924 11:34:23.619807 (+ 1585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:23.619828 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:23.620270 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:23.620693 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:23.621213 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:23.622513 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:23.623847 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:23.624498 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:23.624682 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:23.625827 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:23.625835 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:23.625847 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:34:23.626389 (+ 542us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:23.626456 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:34:23.626479 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:23.626599 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:23.626804 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 11:34:23.627086 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":331,"threads_started":1}
I20260924 11:34:23.628057 12440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P badb2f44bbdc426dafa858e16d238c49: Bootstrap starting.
I20260924 11:34:23.631469 12440 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P badb2f44bbdc426dafa858e16d238c49: Neither blocks nor log segments found. Creating new log.
I20260924 11:34:23.631677 6992 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:34:23.632483 12440 log.cc:824] T 00000000000000000000000000000000 P badb2f44bbdc426dafa858e16d238c49: Log is configured to *not* fsync() on all Append() calls
I20260924 11:34:23.635227 12440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P badb2f44bbdc426dafa858e16d238c49: No bootstrap required, opened a new log
I20260924 11:34:23.642683 12440 raft_consensus.cc:348] T 00000000000000000000000000000000 P badb2f44bbdc426dafa858e16d238c49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "badb2f44bbdc426dafa858e16d238c49" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36247 } }
I20260924 11:34:23.643033 12440 raft_consensus.cc:374] T 00000000000000000000000000000000 P badb2f44bbdc426dafa858e16d238c49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:34:23.643154 12440 raft_consensus.cc:729] T 00000000000000000000000000000000 P badb2f44bbdc426dafa858e16d238c49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: badb2f44bbdc426dafa858e16d238c49, State: Initialized, Role: FOLLOWER
I20260924 11:34:23.643762 12440 consensus_queue.cc:260] T 00000000000000000000000000000000 P badb2f44bbdc426dafa858e16d238c49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "badb2f44bbdc426dafa858e16d238c49" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36247 } }
I20260924 11:34:23.643996 12440 raft_consensus.cc:388] T 00000000000000000000000000000000 P badb2f44bbdc426dafa858e16d238c49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:34:23.644147 12440 raft_consensus.cc:482] T 00000000000000000000000000000000 P badb2f44bbdc426dafa858e16d238c49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:34:23.644323 12440 raft_consensus.cc:3037] T 00000000000000000000000000000000 P badb2f44bbdc426dafa858e16d238c49 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:34:23.646347 12440 raft_consensus.cc:504] T 00000000000000000000000000000000 P badb2f44bbdc426dafa858e16d238c49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "badb2f44bbdc426dafa858e16d238c49" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36247 } }
I20260924 11:34:23.646831 12440 leader_election.cc:302] T 00000000000000000000000000000000 P badb2f44bbdc426dafa858e16d238c49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: badb2f44bbdc426dafa858e16d238c49; no voters:
I20260924 11:34:23.647423 12440 leader_election.cc:288] T 00000000000000000000000000000000 P badb2f44bbdc426dafa858e16d238c49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:34:23.647568 12445 raft_consensus.cc:2781] T 00000000000000000000000000000000 P badb2f44bbdc426dafa858e16d238c49 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:34:23.648257 12445 raft_consensus.cc:686] T 00000000000000000000000000000000 P badb2f44bbdc426dafa858e16d238c49 [term 1 LEADER]: Becoming Leader. State: Replica: badb2f44bbdc426dafa858e16d238c49, State: Running, Role: LEADER
I20260924 11:34:23.648880 12445 consensus_queue.cc:237] T 00000000000000000000000000000000 P badb2f44bbdc426dafa858e16d238c49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "badb2f44bbdc426dafa858e16d238c49" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36247 } }
I20260924 11:34:23.649601 12440 sys_catalog.cc:564] T 00000000000000000000000000000000 P badb2f44bbdc426dafa858e16d238c49 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:34:23.651734 12447 sys_catalog.cc:455] T 00000000000000000000000000000000 P badb2f44bbdc426dafa858e16d238c49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "badb2f44bbdc426dafa858e16d238c49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "badb2f44bbdc426dafa858e16d238c49" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36247 } } }
I20260924 11:34:23.652230 12447 sys_catalog.cc:458] T 00000000000000000000000000000000 P badb2f44bbdc426dafa858e16d238c49 [sys.catalog]: This master's current role is: LEADER
I20260924 11:34:23.652150 12446 sys_catalog.cc:455] T 00000000000000000000000000000000 P badb2f44bbdc426dafa858e16d238c49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader badb2f44bbdc426dafa858e16d238c49. Latest consensus state: current_term: 1 leader_uuid: "badb2f44bbdc426dafa858e16d238c49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "badb2f44bbdc426dafa858e16d238c49" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36247 } } }
I20260924 11:34:23.652467 12446 sys_catalog.cc:458] T 00000000000000000000000000000000 P badb2f44bbdc426dafa858e16d238c49 [sys.catalog]: This master's current role is: LEADER
I20260924 11:34:23.656582 12452 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:34:23.661988 12452 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:34:23.670935 12452 catalog_manager.cc:1347] Generated new cluster ID: 2c20c344bb5c4cda99a5036a408e2169
I20260924 11:34:23.671056 12452 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:34:23.686152 12452 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:34:23.687433 12452 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:34:23.699333 12452 catalog_manager.cc:5868] T 00000000000000000000000000000000 P badb2f44bbdc426dafa858e16d238c49: Generated new TSK 0
I20260924 11:34:23.700258 12452 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:34:23.730840 12443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:23.722472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42092 (local address 127.6.212.62:36247)
0924 11:34:23.722666 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:23.722673 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:23.723168 (+ 495us) server_negotiation.cc:407] Connection header received
0924 11:34:23.723343 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:23.723354 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:23.723473 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:23.723634 (+ 161us) server_negotiation.cc:226] Negotiated authn=JWT
0924 11:34:23.724058 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:23.725174 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:23.726451 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:23.727132 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:23.727247 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:23.727802 (+ 555us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 11:34:23.730073 (+ 2271us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46275/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 11:34:23.730534 (+ 461us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42092: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46275/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":62}
W20260924 11:34:23.730960 12443 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42092: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46275/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 11:34:23.732826 12469 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:34:23.722448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36247 (local address 127.0.0.1:42092)
0924 11:34:23.723069 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:23.723085 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:23.723200 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:23.723675 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:23.723678 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:23.723687 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0924 11:34:23.723921 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:23.723929 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:23.725302 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:23.725305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:23.726324 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:23.726334 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:23.727307 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:23.727311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:23.727504 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:23.727535 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 11:34:23.730281 (+ 2746us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46275/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 11:34:23.732399 (+ 2118us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.212.62:36247: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46275/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":1810,"thread_start_us":362,"threads_started":1}
I20260924 11:34:23.735420 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 12374
I20260924 11:34:23.736568 12435 generic_service.cc:196] Checking for leaks (request via RPC)
E20260924 11:34:23.743468 12470 webserver.cc:533] Webserver: sslize failed:
E20260924 11:34:23.743551 12470 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 11:34:23.827260 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 12374
2026-09-24T11:34:23Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (715 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T11:34:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:34:23Z Disabled control of system clock
I20260924 11:34:23.908560 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:35809
--webserver_interface=127.6.212.62
--webserver_port=0
--builtin_ntp_servers=127.6.212.20:40575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:35809
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790249617057452-6992-0/testchainca.cert with env {}
W20260924 11:34:24.241367 12475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:24.241729 12475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:24.241963 12475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:24.249804 12475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:34:24.249934 12475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:24.250048 12475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:24.250151 12475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:34:24.250226 12475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 11:34:24.250331 12475 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 11:34:24.250415 12475 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 11:34:24.260656 12475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:40575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:35809
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:35809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.212.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790249617057452-6992-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:34:24.262110 12475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:24.263490 12475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:24.275837 12480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:24.275806 12481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:24.276182 12483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:24.276801 12475 server_base.cc:1034] running on GCE node
I20260924 11:34:24.277881 12475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:24.279692 12475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:24.280951 12475 hybrid_clock.cc:648] HybridClock initialized: now 1790249664280898 us; error 56 us; skew 500 ppm
I20260924 11:34:24.281339 12475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:34:24.288878 12475 webserver.cc:460] Webserver started at http://127.6.212.62:35395/ using document root <none> and password file <none>
I20260924 11:34:24.289655 12475 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:24.289800 12475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:24.290131 12475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:34:24.293111 12475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "bb1c351162cf49f194609078080ee41b"
format_stamp: "Formatted at 2026-09-24 11:34:24 on dist-test-slave-cq7c"
server_key: "70997884b38c2f5c551019d092beb846"
server_key_iv: "3ad3db79834929f96a986aa13049650e"
server_key_version: "encryptionkey@0"
I20260924 11:34:24.293929 12475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "bb1c351162cf49f194609078080ee41b"
format_stamp: "Formatted at 2026-09-24 11:34:24 on dist-test-slave-cq7c"
server_key: "70997884b38c2f5c551019d092beb846"
server_key_iv: "3ad3db79834929f96a986aa13049650e"
server_key_version: "encryptionkey@0"
I20260924 11:34:24.299005 12475 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 11:34:24.302446 12489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:24.303925 12475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260924 11:34:24.304116 12475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "bb1c351162cf49f194609078080ee41b"
format_stamp: "Formatted at 2026-09-24 11:34:24 on dist-test-slave-cq7c"
server_key: "70997884b38c2f5c551019d092beb846"
server_key_iv: "3ad3db79834929f96a986aa13049650e"
server_key_version: "encryptionkey@0"
I20260924 11:34:24.304327 12475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:24.321074 12475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:24.321965 12475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:34:24.322291 12475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:24.341840 12475 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.62:35809
I20260924 11:34:24.341818 12540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.62:35809 every 8 connection(s)
I20260924 11:34:24.343263 12475 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:34:24.346920 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 12475
I20260924 11:34:24.347142 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:34:24.347585 12541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:34:24.347980 6992 external_mini_cluster.cc:1428] Setting key 5ab352ae99a605767f3a33fab894926c
I20260924 11:34:24.361105 12541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb1c351162cf49f194609078080ee41b: Bootstrap starting.
I20260924 11:34:24.361219 12544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:24.349867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38124 (local address 127.6.212.62:35809)
0924 11:34:24.350827 (+ 960us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:24.350840 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:24.350878 (+ 38us) server_negotiation.cc:407] Connection header received
0924 11:34:24.352547 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:24.352567 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:24.353021 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:24.353357 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:24.353903 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:24.355208 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:24.356955 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:24.357595 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:24.357723 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:24.358911 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:24.358918 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:24.358931 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:34:24.359525 (+ 594us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:24.359598 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:34:24.359617 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:24.359706 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:24.359932 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 11:34:24.360302 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":353,"threads_started":1}
I20260924 11:34:24.364442 12541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb1c351162cf49f194609078080ee41b: Neither blocks nor log segments found. Creating new log.
I20260924 11:34:24.364938 6992 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:34:24.365641 12541 log.cc:824] T 00000000000000000000000000000000 P bb1c351162cf49f194609078080ee41b: Log is configured to *not* fsync() on all Append() calls
I20260924 11:34:24.368552 12541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb1c351162cf49f194609078080ee41b: No bootstrap required, opened a new log
I20260924 11:34:24.375345 12541 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb1c351162cf49f194609078080ee41b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb1c351162cf49f194609078080ee41b" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 35809 } }
I20260924 11:34:24.375764 12541 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb1c351162cf49f194609078080ee41b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:34:24.375856 12541 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb1c351162cf49f194609078080ee41b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb1c351162cf49f194609078080ee41b, State: Initialized, Role: FOLLOWER
I20260924 11:34:24.376526 12541 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb1c351162cf49f194609078080ee41b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb1c351162cf49f194609078080ee41b" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 35809 } }
I20260924 11:34:24.376760 12541 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb1c351162cf49f194609078080ee41b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:34:24.376905 12541 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb1c351162cf49f194609078080ee41b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:34:24.377089 12541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb1c351162cf49f194609078080ee41b [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:34:24.379138 12541 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb1c351162cf49f194609078080ee41b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb1c351162cf49f194609078080ee41b" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 35809 } }
I20260924 11:34:24.379693 12541 leader_election.cc:302] T 00000000000000000000000000000000 P bb1c351162cf49f194609078080ee41b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb1c351162cf49f194609078080ee41b; no voters:
I20260924 11:34:24.380290 12541 leader_election.cc:288] T 00000000000000000000000000000000 P bb1c351162cf49f194609078080ee41b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:34:24.380434 12546 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb1c351162cf49f194609078080ee41b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:34:24.381471 12546 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb1c351162cf49f194609078080ee41b [term 1 LEADER]: Becoming Leader. State: Replica: bb1c351162cf49f194609078080ee41b, State: Running, Role: LEADER
I20260924 11:34:24.382169 12546 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb1c351162cf49f194609078080ee41b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb1c351162cf49f194609078080ee41b" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 35809 } }
I20260924 11:34:24.382695 12541 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb1c351162cf49f194609078080ee41b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:34:24.385152 12548 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb1c351162cf49f194609078080ee41b [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb1c351162cf49f194609078080ee41b. Latest consensus state: current_term: 1 leader_uuid: "bb1c351162cf49f194609078080ee41b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb1c351162cf49f194609078080ee41b" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 35809 } } }
I20260924 11:34:24.385509 12548 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb1c351162cf49f194609078080ee41b [sys.catalog]: This master's current role is: LEADER
I20260924 11:34:24.385249 12547 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb1c351162cf49f194609078080ee41b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb1c351162cf49f194609078080ee41b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb1c351162cf49f194609078080ee41b" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 35809 } } }
I20260924 11:34:24.385653 12547 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb1c351162cf49f194609078080ee41b [sys.catalog]: This master's current role is: LEADER
I20260924 11:34:24.391563 12551 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:34:24.397516 12551 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:34:24.405901 12551 catalog_manager.cc:1347] Generated new cluster ID: 5549fead21764d06ba76f240afc98997
I20260924 11:34:24.406065 12551 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:34:24.422230 12551 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:34:24.424281 12551 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:34:24.437706 12551 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb1c351162cf49f194609078080ee41b: Generated new TSK 0
I20260924 11:34:24.438906 12551 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:34:24.514514 12544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:24.450359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38138 (local address 127.6.212.62:35809)
0924 11:34:24.450535 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:24.450542 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:24.450928 (+ 386us) server_negotiation.cc:407] Connection header received
0924 11:34:24.451135 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:24.451141 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:24.451246 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:24.451462 (+ 216us) server_negotiation.cc:226] Negotiated authn=JWT
0924 11:34:24.451937 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:24.453062 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:24.454479 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:24.455173 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:24.455302 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:24.455840 (+ 538us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 11:34:24.513790 (+ 57950us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 11:34:24.514190 (+ 400us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38138: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":66}
W20260924 11:34:24.514655 12544 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38138: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 11:34:24.514806 12570 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:34:24.450218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35809 (local address 127.0.0.1:38138)
0924 11:34:24.450820 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:24.450837 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:24.450951 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:24.451469 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:24.451473 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:24.451483 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0924 11:34:24.451782 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:24.451791 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:24.453223 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:24.453227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:24.454325 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:24.454336 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:24.455334 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:24.455339 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:24.455545 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:24.455578 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 11:34:24.514098 (+ 58520us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 11:34:24.514342 (+ 244us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.212.62:35809: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":328,"threads_started":1}
I20260924 11:34:24.516222 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 12475
I20260924 11:34:24.517416 12536 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:34:24.614480 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 12475
2026-09-24T11:34:24Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (767 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (409 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (419 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:34:25 dist-test-slave-cq7c krb5kdc[12585](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:34:25 dist-test-slave-cq7c krb5kdc[12585](info): set up 2 sockets
Sep 24 11:34:25 dist-test-slave-cq7c krb5kdc[12585](info): commencing operation
krb5kdc: starting...
W20260924 11:34:26.576064 6992 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.010s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Sep 24 11:34:26 dist-test-slave-cq7c krb5kdc[12585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:34:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:34:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.212.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
I20260924 11:34:26.796131 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:38935
--webserver_interface=127.6.212.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:34619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:38935
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:34:27.128012 12598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:27.128371 12598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:27.128538 12598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:27.135890 12598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:34:27.136058 12598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:27.136164 12598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:27.136274 12598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:34:27.136368 12598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:34:27.146344 12598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:34619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:38935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:38935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.212.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:34:27.147766 12598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:27.149120 12598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:27.161376 12603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:27.161289 12606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:27.161531 12604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:27.162756 12598 server_base.cc:1034] running on GCE node
I20260924 11:34:27.163702 12598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:27.165203 12598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:27.166496 12598 hybrid_clock.cc:648] HybridClock initialized: now 1790249667166497 us; error 115 us; skew 500 ppm
I20260924 11:34:27.166889 12598 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:34:27 dist-test-slave-cq7c krb5kdc[12585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249667, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:27.170053 12598 init.cc:377] Logged in from keytab as kudu/127.6.212.62@KRBTEST.COM (short username kudu)
I20260924 11:34:27.172519 12598 webserver.cc:460] Webserver started at http://127.6.212.62:36935/ using document root <none> and password file <none>
I20260924 11:34:27.173233 12598 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:27.173386 12598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:27.173705 12598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:34:27.176851 12598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "5044654d3887446bb1784f2385d563f8"
format_stamp: "Formatted at 2026-09-24 11:34:27 on dist-test-slave-cq7c"
server_key: "7b10073331e6ca3415492d11f94f6538"
server_key_iv: "e5dfcc3c80228bedfc9fe23df216fc35"
server_key_version: "encryptionkey@0"
I20260924 11:34:27.177676 12598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "5044654d3887446bb1784f2385d563f8"
format_stamp: "Formatted at 2026-09-24 11:34:27 on dist-test-slave-cq7c"
server_key: "7b10073331e6ca3415492d11f94f6538"
server_key_iv: "e5dfcc3c80228bedfc9fe23df216fc35"
server_key_version: "encryptionkey@0"
I20260924 11:34:27.183084 12598 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 11:34:27.186781 12613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:27.188377 12598 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 11:34:27.188598 12598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "5044654d3887446bb1784f2385d563f8"
format_stamp: "Formatted at 2026-09-24 11:34:27 on dist-test-slave-cq7c"
server_key: "7b10073331e6ca3415492d11f94f6538"
server_key_iv: "e5dfcc3c80228bedfc9fe23df216fc35"
server_key_version: "encryptionkey@0"
I20260924 11:34:27.188814 12598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:27.208125 12598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:27.212467 12598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:34:27.212800 12598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:27.232532 12598 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.62:38935
I20260924 11:34:27.232571 12664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.62:38935 every 8 connection(s)
I20260924 11:34:27.233955 12598 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:34:27.236961 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 12598
I20260924 11:34:27.237197 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:34:27.237816 6992 external_mini_cluster.cc:1428] Setting key 513a2d191bcce01e3f63073bd3654f12
I20260924 11:34:27.237996 12665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:34:27.252671 12665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5044654d3887446bb1784f2385d563f8: Bootstrap starting.
Sep 24 11:34:27 dist-test-slave-cq7c krb5kdc[12585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:34:27.256757 12665 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5044654d3887446bb1784f2385d563f8: Neither blocks nor log segments found. Creating new log.
I20260924 11:34:27.258121 12665 log.cc:824] T 00000000000000000000000000000000 P 5044654d3887446bb1784f2385d563f8: Log is configured to *not* fsync() on all Append() calls
I20260924 11:34:27.261695 12665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5044654d3887446bb1784f2385d563f8: No bootstrap required, opened a new log
I20260924 11:34:27.262955 12668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:27.239901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35194 (local address 127.6.212.62:38935)
0924 11:34:27.240714 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:27.240730 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:27.240785 (+ 55us) server_negotiation.cc:407] Connection header received
0924 11:34:27.243022 (+ 2237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:27.243054 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:27.243656 (+ 602us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:27.244141 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:27.245511 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:27.247484 (+ 1973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:27.248931 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:27.250033 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:27.250221 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:27.254340 (+ 4119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:27.254352 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:27.254372 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:27.254429 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:27.258344 (+ 3915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:27.259195 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:27.259201 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:27.259207 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:27.259430 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:27.259838 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:27.259843 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:27.259846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:27.260488 (+ 642us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:27.260759 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:27.261155 (+ 396us) server_negotiation.cc:299] Negotiation successful
0924 11:34:27.261574 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":325,"threads_started":1}
I20260924 11:34:27.270023 12665 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5044654d3887446bb1784f2385d563f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5044654d3887446bb1784f2385d563f8" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 38935 } }
I20260924 11:34:27.270355 12665 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5044654d3887446bb1784f2385d563f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:34:27.270418 12665 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5044654d3887446bb1784f2385d563f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5044654d3887446bb1784f2385d563f8, State: Initialized, Role: FOLLOWER
I20260924 11:34:27.271051 12665 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5044654d3887446bb1784f2385d563f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5044654d3887446bb1784f2385d563f8" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 38935 } }
I20260924 11:34:27.271405 12665 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5044654d3887446bb1784f2385d563f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:34:27.271586 12665 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5044654d3887446bb1784f2385d563f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:34:27.271772 12665 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5044654d3887446bb1784f2385d563f8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:34:27.273898 12665 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5044654d3887446bb1784f2385d563f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5044654d3887446bb1784f2385d563f8" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 38935 } }
I20260924 11:34:27.274403 12665 leader_election.cc:302] T 00000000000000000000000000000000 P 5044654d3887446bb1784f2385d563f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5044654d3887446bb1784f2385d563f8; no voters:
I20260924 11:34:27.275113 12665 leader_election.cc:288] T 00000000000000000000000000000000 P 5044654d3887446bb1784f2385d563f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:34:27.275273 12670 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5044654d3887446bb1784f2385d563f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:34:27.276649 12670 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5044654d3887446bb1784f2385d563f8 [term 1 LEADER]: Becoming Leader. State: Replica: 5044654d3887446bb1784f2385d563f8, State: Running, Role: LEADER
I20260924 11:34:27.277156 12670 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5044654d3887446bb1784f2385d563f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5044654d3887446bb1784f2385d563f8" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 38935 } }
I20260924 11:34:27.277302 12665 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5044654d3887446bb1784f2385d563f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:34:27.280185 12671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5044654d3887446bb1784f2385d563f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5044654d3887446bb1784f2385d563f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5044654d3887446bb1784f2385d563f8" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 38935 } } }
I20260924 11:34:27.280154 12672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5044654d3887446bb1784f2385d563f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5044654d3887446bb1784f2385d563f8. Latest consensus state: current_term: 1 leader_uuid: "5044654d3887446bb1784f2385d563f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5044654d3887446bb1784f2385d563f8" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 38935 } } }
I20260924 11:34:27.280656 12672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5044654d3887446bb1784f2385d563f8 [sys.catalog]: This master's current role is: LEADER
I20260924 11:34:27.280656 12671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5044654d3887446bb1784f2385d563f8 [sys.catalog]: This master's current role is: LEADER
I20260924 11:34:27.284824 12678 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:34:27.290915 12678 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:34:27.299823 12678 catalog_manager.cc:1347] Generated new cluster ID: e478e247ee0c485fabea57b27242600e
I20260924 11:34:27.299939 12678 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:34:27.306244 12678 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:34:27.308163 12678 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:34:27.322518 12678 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5044654d3887446bb1784f2385d563f8: Generated new TSK 0
I20260924 11:34:27.323697 12678 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.212.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab.
Entry for principal kudu/127.6.212.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab.
I20260924 11:34:27.384440 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.1:0
--local_ip_for_outbound_sockets=127.6.212.1
--webserver_interface=127.6.212.1
--webserver_port=0
--tserver_master_addrs=127.6.212.62:38935
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab
--principal=kudu/127.6.212.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:34619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:34:27.718664 12691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:27.718995 12691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:27.719151 12691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:27.726447 12691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:27.726606 12691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:27.726785 12691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.1
I20260924 11:34:27.736876 12691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:34619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab
--principal=kudu/127.6.212.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.212.1
--webserver_port=0
--tserver_master_addrs=127.6.212.62:38935
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.1
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:34:27.738298 12691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:27.739710 12691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:27.753537 12697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:27.753504 12699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:27.753623 12696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:27.754686 12691 server_base.cc:1034] running on GCE node
I20260924 11:34:27.755342 12691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:27.756456 12691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:27.757665 12691 hybrid_clock.cc:648] HybridClock initialized: now 1790249667757616 us; error 45 us; skew 500 ppm
I20260924 11:34:27.758006 12691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:34:27 dist-test-slave-cq7c krb5kdc[12585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249667, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:27.761616 12691 init.cc:377] Logged in from keytab as kudu/127.6.212.1@KRBTEST.COM (short username kudu)
I20260924 11:34:27.763579 12691 webserver.cc:460] Webserver started at http://127.6.212.1:34951/ using document root <none> and password file <none>
I20260924 11:34:27.764375 12691 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:27.764536 12691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:27.764914 12691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:34:27.768417 12691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-0/data/instance:
uuid: "33edf80a3e44490a8a8eb5a1acfb6035"
format_stamp: "Formatted at 2026-09-24 11:34:27 on dist-test-slave-cq7c"
server_key: "b086504631ffcd83c603fb385f37f75f"
server_key_iv: "abd88f4ad79bec3e8c16f2d6a1f161ad"
server_key_version: "encryptionkey@0"
I20260924 11:34:27.769273 12691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance:
uuid: "33edf80a3e44490a8a8eb5a1acfb6035"
format_stamp: "Formatted at 2026-09-24 11:34:27 on dist-test-slave-cq7c"
server_key: "b086504631ffcd83c603fb385f37f75f"
server_key_iv: "abd88f4ad79bec3e8c16f2d6a1f161ad"
server_key_version: "encryptionkey@0"
I20260924 11:34:27.774601 12691 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 11:34:27.778259 12706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:27.779837 12691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:34:27.779978 12691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-0/wal
uuid: "33edf80a3e44490a8a8eb5a1acfb6035"
format_stamp: "Formatted at 2026-09-24 11:34:27 on dist-test-slave-cq7c"
server_key: "b086504631ffcd83c603fb385f37f75f"
server_key_iv: "abd88f4ad79bec3e8c16f2d6a1f161ad"
server_key_version: "encryptionkey@0"
I20260924 11:34:27.780208 12691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:27.804996 12691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:27.809094 12691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:34:27.809399 12691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:27.810362 12691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:34:27.811735 12691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:34:27.811844 12691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:27.811995 12691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:34:27.812078 12691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:27.844895 12691 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.1:35777
I20260924 11:34:27.844987 12818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.1:35777 every 8 connection(s)
I20260924 11:34:27.846221 12691 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
I20260924 11:34:27.847842 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 12691
I20260924 11:34:27.848098 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance
I20260924 11:34:27.848909 6992 external_mini_cluster.cc:1428] Setting key 9aac7a6c1bd5e7a9ec29d112751ddd75
Sep 24 11:34:27 dist-test-slave-cq7c krb5kdc[12585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249667, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:34:27.869946 12821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:27.848821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:52413 (local address 127.6.212.62:38935)
0924 11:34:27.849264 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:27.849268 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:27.850491 (+ 1223us) server_negotiation.cc:407] Connection header received
0924 11:34:27.853181 (+ 2690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:27.853186 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:27.853272 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:27.853404 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:27.855542 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:27.856301 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:27.858727 (+ 2426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:27.859124 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:27.859269 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:27.862973 (+ 3704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:27.862984 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:27.862992 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:27.863038 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:27.866538 (+ 3500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:27.867300 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:27.867309 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:27.867316 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:27.867509 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:27.868001 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:27.868009 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:27.868015 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:27.868415 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:27.868582 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:27.869283 (+ 701us) server_negotiation.cc:299] Negotiation successful
0924 11:34:27.869536 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":246,"threads_started":1}
I20260924 11:34:27.870651 12822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:27.849500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:38935 (local address 127.6.212.1:52413)
0924 11:34:27.850345 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:27.850380 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:27.852840 (+ 2460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:27.853677 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:27.853691 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:27.854490 (+ 799us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:27.855334 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:27.855388 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:27.856458 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:27.856466 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:27.858556 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:27.858568 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:27.859386 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:27.859395 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:27.859639 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:27.860351 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:27.860374 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:27.862737 (+ 2363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:27.866692 (+ 3955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:27.866699 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:27.866714 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:27.867156 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:27.867646 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:27.867651 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:27.867655 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:27.867865 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:27.868543 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:27.868549 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:27.868878 (+ 329us) client_negotiation.cc:769] Sending connection context
0924 11:34:27.869114 (+ 236us) client_negotiation.cc:240] Negotiation successful
0924 11:34:27.869504 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":402,"threads_started":1}
WARNING: no policy specified for kudu/127.6.212.2@KRBTEST.COM; defaulting to no policy
I20260924 11:34:27.872354 12819 heartbeater.cc:347] Connected to a master server at 127.6.212.62:38935
I20260924 11:34:27.872671 12819 heartbeater.cc:464] Registering TS with master...
I20260924 11:34:27.873462 12819 heartbeater.cc:511] Master 127.6.212.62:38935 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.2@KRBTEST.COM" created.
I20260924 11:34:27.876154 12630 ts_manager.cc:194] Registered new tserver with Master: 33edf80a3e44490a8a8eb5a1acfb6035 (127.6.212.1:35777)
I20260924 11:34:27.878643 12630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.1@KRBTEST.COM'} at 127.6.212.1:52413
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab.
Entry for principal kudu/127.6.212.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab.
I20260924 11:34:27.903965 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.2:0
--local_ip_for_outbound_sockets=127.6.212.2
--webserver_interface=127.6.212.2
--webserver_port=0
--tserver_master_addrs=127.6.212.62:38935
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab
--principal=kudu/127.6.212.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:34619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:34:28.226137 12825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:28.226426 12825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:28.226599 12825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:28.234107 12825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:28.234270 12825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:28.234444 12825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.2
I20260924 11:34:28.244510 12825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:34619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab
--principal=kudu/127.6.212.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.212.2
--webserver_port=0
--tserver_master_addrs=127.6.212.62:38935
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.2
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:34:28.245862 12825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:28.247216 12825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:28.260473 12830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:28.260516 12831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:28.260439 12833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:28.261262 12825 server_base.cc:1034] running on GCE node
I20260924 11:34:28.261837 12825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:28.262738 12825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:28.263971 12825 hybrid_clock.cc:648] HybridClock initialized: now 1790249668263923 us; error 63 us; skew 500 ppm
I20260924 11:34:28.264305 12825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:34:28 dist-test-slave-cq7c krb5kdc[12585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249668, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:28.267477 12825 init.cc:377] Logged in from keytab as kudu/127.6.212.2@KRBTEST.COM (short username kudu)
I20260924 11:34:28.268873 12825 webserver.cc:460] Webserver started at http://127.6.212.2:37463/ using document root <none> and password file <none>
I20260924 11:34:28.269550 12825 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:28.269696 12825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:28.269997 12825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:34:28.272923 12825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-1/data/instance:
uuid: "4171200ef41a468e9b8d27c4d79c80b3"
format_stamp: "Formatted at 2026-09-24 11:34:28 on dist-test-slave-cq7c"
server_key: "1915d7ebd65eb26dddc20cbfb348fa04"
server_key_iv: "f47fb337312d84f86efbdc5167f0915f"
server_key_version: "encryptionkey@0"
I20260924 11:34:28.273695 12825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance:
uuid: "4171200ef41a468e9b8d27c4d79c80b3"
format_stamp: "Formatted at 2026-09-24 11:34:28 on dist-test-slave-cq7c"
server_key: "1915d7ebd65eb26dddc20cbfb348fa04"
server_key_iv: "f47fb337312d84f86efbdc5167f0915f"
server_key_version: "encryptionkey@0"
I20260924 11:34:28.279027 12825 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 11:34:28.282776 12840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:28.284742 12825 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260924 11:34:28.284909 12825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-1/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-1/wal
uuid: "4171200ef41a468e9b8d27c4d79c80b3"
format_stamp: "Formatted at 2026-09-24 11:34:28 on dist-test-slave-cq7c"
server_key: "1915d7ebd65eb26dddc20cbfb348fa04"
server_key_iv: "f47fb337312d84f86efbdc5167f0915f"
server_key_version: "encryptionkey@0"
I20260924 11:34:28.285153 12825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:28.310441 12825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:28.314898 12825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:34:28.315230 12825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:28.316321 12825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:34:28.317698 12825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:34:28.317803 12825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:28.317955 12825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:34:28.318038 12825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:28.350479 12825 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.2:43751
I20260924 11:34:28.350490 12952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.2:43751 every 8 connection(s)
I20260924 11:34:28.351840 12825 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
I20260924 11:34:28.355077 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 12825
I20260924 11:34:28.355259 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance
I20260924 11:34:28.355732 6992 external_mini_cluster.cc:1428] Setting key 333ffdc1fc749847f7e826959962d02e
Sep 24 11:34:28 dist-test-slave-cq7c krb5kdc[12585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249668, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.2@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:34:28.374466 12821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:28.354562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:43175 (local address 127.6.212.62:38935)
0924 11:34:28.354776 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:28.354782 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:28.356143 (+ 1361us) server_negotiation.cc:407] Connection header received
0924 11:34:28.358119 (+ 1976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:28.358124 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:28.358209 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:28.358321 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:28.359888 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:28.361019 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:28.362679 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:28.363472 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:28.363623 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:28.367071 (+ 3448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:28.367080 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:28.367087 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:28.367133 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:28.370912 (+ 3779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:28.371822 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:28.371831 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:28.371837 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:28.372001 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:28.372429 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:28.372437 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:28.372443 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:28.372841 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:28.372995 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:28.373740 (+ 745us) server_negotiation.cc:299] Negotiation successful
0924 11:34:28.374050 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 11:34:28.375200 12955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:28.355045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:38935 (local address 127.6.212.2:43175)
0924 11:34:28.355950 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:28.356004 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:28.357846 (+ 1842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:28.358551 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:28.358560 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:28.359054 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:28.359717 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:28.359737 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:28.361175 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:28.361179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:28.362518 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:28.362530 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:28.363687 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:28.363692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:28.363900 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:28.364632 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:28.364654 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:28.366825 (+ 2171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:28.371088 (+ 4263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:28.371096 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:28.371111 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:28.371687 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:28.372117 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:28.372120 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:28.372122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:28.372296 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:28.372965 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:28.372970 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:28.373325 (+ 355us) client_negotiation.cc:769] Sending connection context
0924 11:34:28.373561 (+ 236us) client_negotiation.cc:240] Negotiation successful
0924 11:34:28.373968 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":417,"threads_started":1}
I20260924 11:34:28.376900 12953 heartbeater.cc:347] Connected to a master server at 127.6.212.62:38935
I20260924 11:34:28.377264 12953 heartbeater.cc:464] Registering TS with master...
I20260924 11:34:28.378283 12953 heartbeater.cc:511] Master 127.6.212.62:38935 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.212.3@KRBTEST.COM; defaulting to no policy
I20260924 11:34:28.380125 12630 ts_manager.cc:194] Registered new tserver with Master: 4171200ef41a468e9b8d27c4d79c80b3 (127.6.212.2:43751)
I20260924 11:34:28.380973 12630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.2@KRBTEST.COM'} at 127.6.212.2:43175
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab.
Entry for principal kudu/127.6.212.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab.
I20260924 11:34:28.410743 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.3:0
--local_ip_for_outbound_sockets=127.6.212.3
--webserver_interface=127.6.212.3
--webserver_port=0
--tserver_master_addrs=127.6.212.62:38935
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab
--principal=kudu/127.6.212.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:34619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:34:28.742835 12958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:28.743216 12958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:28.743469 12958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:28.751099 12958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:28.751291 12958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:28.751526 12958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.3
I20260924 11:34:28.762274 12958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:34619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab
--principal=kudu/127.6.212.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.212.3
--webserver_port=0
--tserver_master_addrs=127.6.212.62:38935
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.3
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:34:28.763731 12958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:28.765079 12958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:28.778419 12966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:28.778514 12963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:28.778514 12964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:28.779788 12958 server_base.cc:1034] running on GCE node
I20260924 11:34:28.780478 12958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:28.781467 12958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:28.782677 12958 hybrid_clock.cc:648] HybridClock initialized: now 1790249668782607 us; error 72 us; skew 500 ppm
I20260924 11:34:28.783066 12958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:34:28 dist-test-slave-cq7c krb5kdc[12585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249668, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:28.786650 12958 init.cc:377] Logged in from keytab as kudu/127.6.212.3@KRBTEST.COM (short username kudu)
I20260924 11:34:28.788223 12958 webserver.cc:460] Webserver started at http://127.6.212.3:42999/ using document root <none> and password file <none>
I20260924 11:34:28.789011 12958 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:28.789167 12958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:28.789486 12958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:34:28.792743 12958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-2/data/instance:
uuid: "cdc77725263c4e2ca65a2b667c45cbc6"
format_stamp: "Formatted at 2026-09-24 11:34:28 on dist-test-slave-cq7c"
server_key: "c5e99be68880891986635b0db803ed9f"
server_key_iv: "28f1716aeebf7611b2bde444c8a337f8"
server_key_version: "encryptionkey@0"
I20260924 11:34:28.793632 12958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance:
uuid: "cdc77725263c4e2ca65a2b667c45cbc6"
format_stamp: "Formatted at 2026-09-24 11:34:28 on dist-test-slave-cq7c"
server_key: "c5e99be68880891986635b0db803ed9f"
server_key_iv: "28f1716aeebf7611b2bde444c8a337f8"
server_key_version: "encryptionkey@0"
I20260924 11:34:28.798925 12958 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 11:34:28.802713 12973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:28.804225 12958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:34:28.804412 12958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-2/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-2/wal
uuid: "cdc77725263c4e2ca65a2b667c45cbc6"
format_stamp: "Formatted at 2026-09-24 11:34:28 on dist-test-slave-cq7c"
server_key: "c5e99be68880891986635b0db803ed9f"
server_key_iv: "28f1716aeebf7611b2bde444c8a337f8"
server_key_version: "encryptionkey@0"
I20260924 11:34:28.804646 12958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:28.838414 12958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:28.842672 12958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:34:28.842981 12958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:28.844089 12958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:34:28.845611 12958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:34:28.845721 12958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:28.845886 12958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:34:28.845961 12958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:28.880643 12958 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.3:36539
I20260924 11:34:28.880688 13085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.3:36539 every 8 connection(s)
I20260924 11:34:28.881982 12958 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
I20260924 11:34:28.882651 12819 heartbeater.cc:503] Master 127.6.212.62:38935 was elected leader, sending a full tablet report...
I20260924 11:34:28.891850 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 12958
I20260924 11:34:28.892097 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance
I20260924 11:34:28.892581 6992 external_mini_cluster.cc:1428] Setting key efc3b1cca2aaa333ac4971279229c7b5
Sep 24 11:34:28 dist-test-slave-cq7c krb5kdc[12585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249668, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.3@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:34:28.904594 13088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:28.885085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:35785 (local address 127.6.212.62:38935)
0924 11:34:28.885504 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:28.885510 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:28.886796 (+ 1286us) server_negotiation.cc:407] Connection header received
0924 11:34:28.888615 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:28.888619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:28.888692 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:28.888869 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:28.890431 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:28.891280 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:28.893449 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:28.894118 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:28.894242 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:28.898467 (+ 4225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:28.898472 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:28.898475 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:28.898509 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:28.901156 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:28.902006 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:28.902014 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:28.902020 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:28.902152 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:28.902618 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:28.902631 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:28.902635 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:28.902923 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:28.903097 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:28.903987 (+ 890us) server_negotiation.cc:299] Negotiation successful
0924 11:34:28.904216 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":202,"threads_started":1}
I20260924 11:34:28.905603 13089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:28.885919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:38935 (local address 127.6.212.3:35785)
0924 11:34:28.886638 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:28.886682 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:28.888338 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:28.889147 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:28.889157 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:28.889673 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:28.890242 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:28.890259 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:28.891591 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:28.891595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:28.893236 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:28.893253 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:28.894415 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:28.894422 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:28.894645 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:28.895529 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:28.895569 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:28.898222 (+ 2653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:28.901319 (+ 3097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:28.901329 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:28.901353 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:28.901867 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:28.902286 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:28.902293 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:28.902301 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:28.902488 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:28.903086 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:28.903094 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:28.903555 (+ 461us) client_negotiation.cc:769] Sending connection context
0924 11:34:28.903895 (+ 340us) client_negotiation.cc:240] Negotiation successful
0924 11:34:28.904455 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":325,"threads_started":1}
I20260924 11:34:28.907193 13086 heartbeater.cc:347] Connected to a master server at 127.6.212.62:38935
I20260924 11:34:28.907567 13086 heartbeater.cc:464] Registering TS with master...
I20260924 11:34:28.908591 13086 heartbeater.cc:511] Master 127.6.212.62:38935 requested a full tablet report, sending...
I20260924 11:34:28.910233 12630 ts_manager.cc:194] Registered new tserver with Master: cdc77725263c4e2ca65a2b667c45cbc6 (127.6.212.3:36539)
I20260924 11:34:28.911175 12630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.3@KRBTEST.COM'} at 127.6.212.3:35785
I20260924 11:34:28.921996 6992 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/krb5kdc/test-admin.keytab.
Sep 24 11:34:28 dist-test-slave-cq7c krb5kdc[12585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:28.950121 6992 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 11:34:28.955821 13094 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:34:28.952805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32782 (local address 127.6.212.1:35777)
0924 11:34:28.953802 (+ 997us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:28.953814 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:28.953850 (+ 36us) server_negotiation.cc:407] Connection header received
0924 11:34:28.953928 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:28.953934 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:28.954182 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:28.954363 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:28.955384 (+ 1021us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32782: BlockingRecv error: recv got EOF from 127.0.0.1:32782 (error 108)
Metrics: {"server-negotiator.queue_time_us":816,"thread_start_us":638,"threads_started":1}
W20260924 11:34:28.955972 6992 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/krb5kdc/krb5cc' of type FILE
Sep 24 11:34:28 dist-test-slave-cq7c krb5kdc[12585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:28.957943 6992 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 11:34:28 dist-test-slave-cq7c krb5kdc[12585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:28.959527 6992 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 11:34:28 dist-test-slave-cq7c krb5kdc[12585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.1@KRBTEST.COM
I20260924 11:34:28.974776 13094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:28.961134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32796 (local address 127.6.212.1:35777)
0924 11:34:28.961322 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:28.961330 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:28.961482 (+ 152us) server_negotiation.cc:407] Connection header received
0924 11:34:28.961649 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:28.961653 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:28.961711 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:28.961797 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:28.962697 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:28.963827 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:28.965101 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:28.965684 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:28.965783 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:28.969313 (+ 3530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:28.969325 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:28.969340 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:28.969372 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:28.972279 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:28.972989 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:28.972996 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:28.972998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:28.973149 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:28.973451 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:28.973454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:28.973456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:28.973769 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:28.973989 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:28.974213 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 11:34:28.974477 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260924 11:34:28.980751 13094 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:34:28.979193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32804 (local address 127.6.212.1:35777)
0924 11:34:28.979421 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:28.979427 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:28.979633 (+ 206us) server_negotiation.cc:407] Connection header received
0924 11:34:28.979747 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:28.979750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:28.979804 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:28.979880 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:28.980535 (+ 655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32804: BlockingRecv error: recv got EOF from 127.0.0.1:32804 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260924 11:34:28.981148 6992 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/krb5kdc/krb5cc' of type FILE
Sep 24 11:34:28 dist-test-slave-cq7c krb5kdc[12585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:28.982692 6992 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 11:34:28 dist-test-slave-cq7c krb5kdc[12585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:28.984110 6992 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 11:34:28 dist-test-slave-cq7c krb5kdc[12585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.1@KRBTEST.COM
I20260924 11:34:28.998687 13094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:28.985405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32812 (local address 127.6.212.1:35777)
0924 11:34:28.985573 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:28.985581 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:28.985943 (+ 362us) server_negotiation.cc:407] Connection header received
0924 11:34:28.986148 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:28.986153 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:28.986221 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:28.986322 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:28.987227 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:28.987993 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:28.989265 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:28.989735 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:28.989867 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:28.993474 (+ 3607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:28.993480 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:28.993484 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:28.993538 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:28.996193 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:28.996849 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:28.996857 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:28.996860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:28.996989 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:28.997373 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:28.997378 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:28.997382 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:28.997657 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:28.997926 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:28.998187 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 11:34:28.998386 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260924 11:34:29.003242 13094 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:34:29.001439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32814 (local address 127.6.212.1:35777)
0924 11:34:29.001629 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:29.001647 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:29.002031 (+ 384us) server_negotiation.cc:407] Connection header received
0924 11:34:29.002080 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:29.002083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:29.002149 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:29.002226 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:29.002876 (+ 650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32814: BlockingRecv error: recv got EOF from 127.0.0.1:32814 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
W20260924 11:34:29.003389 6992 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790249617057452-6992-0/krb5kdc/krb5cc' of type FILE
Sep 24 11:34:29 dist-test-slave-cq7c krb5kdc[12585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:29.005041 6992 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 11:34:29 dist-test-slave-cq7c krb5kdc[12585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:29.006508 6992 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 11:34:29 dist-test-slave-cq7c krb5kdc[12585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.1@KRBTEST.COM
I20260924 11:34:29.021937 13094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:29.008174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32828 (local address 127.6.212.1:35777)
0924 11:34:29.008357 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:29.008364 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:29.008524 (+ 160us) server_negotiation.cc:407] Connection header received
0924 11:34:29.008750 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:29.008755 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:29.008820 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:29.008924 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:29.009962 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:29.010679 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:29.012015 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:29.012443 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:29.012586 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:29.016324 (+ 3738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:29.016330 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:29.016333 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:29.016370 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:29.019040 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:29.019819 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:29.019826 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:29.019830 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:29.019984 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:29.020387 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:29.020392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:29.020401 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:29.020702 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:29.021048 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:29.021326 (+ 278us) server_negotiation.cc:299] Negotiation successful
0924 11:34:29.021553 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 11:34:29.023813 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 12691
I20260924 11:34:29.036336 13094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:29.024599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32834 (local address 127.6.212.1:35777)
0924 11:34:29.024784 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:29.024790 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:29.025178 (+ 388us) server_negotiation.cc:407] Connection header received
0924 11:34:29.025370 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:29.025375 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:29.025437 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:29.025552 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:29.026485 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:29.027156 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:29.028390 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:29.028894 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:29.029063 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:29.031207 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:29.031212 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:29.031216 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:29.031245 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:29.033763 (+ 2518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:29.034390 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:29.034395 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:29.034399 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:29.034533 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:29.034898 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:29.034904 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:29.034907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:29.035163 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:29.035466 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:29.035826 (+ 360us) server_negotiation.cc:299] Negotiation successful
0924 11:34:29.036031 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 11:34:29.036939 12814 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:34:29.120729 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 12691
I20260924 11:34:29.139663 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 12825
Sep 24 11:34:29 dist-test-slave-cq7c krb5kdc[12585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.2@KRBTEST.COM
I20260924 11:34:29.156701 13107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:29.140648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44610 (local address 127.6.212.2:43751)
0924 11:34:29.141207 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:29.141213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:29.141240 (+ 27us) server_negotiation.cc:407] Connection header received
0924 11:34:29.141298 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:29.141305 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:29.141515 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:29.141668 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:29.142558 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:29.143704 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:29.145157 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:29.145714 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:29.145820 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:29.150025 (+ 4205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:29.150039 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:29.150056 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:29.150095 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:29.153371 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:29.154315 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:29.154327 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:29.154330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:29.154483 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:29.154985 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:29.154989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:29.154992 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:29.155543 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:29.155790 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:29.156144 (+ 354us) server_negotiation.cc:299] Negotiation successful
0924 11:34:29.156374 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":291,"threads_started":1}
I20260924 11:34:29.158136 12948 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:34:29.244340 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 12825
I20260924 11:34:29.260464 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 12958
Sep 24 11:34:29 dist-test-slave-cq7c krb5kdc[12585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.3@KRBTEST.COM
I20260924 11:34:29.277297 13109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:29.261526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56418 (local address 127.6.212.3:36539)
0924 11:34:29.262238 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:29.262245 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:29.262279 (+ 34us) server_negotiation.cc:407] Connection header received
0924 11:34:29.262347 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:29.262354 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:29.262603 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:29.262813 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:29.264072 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:29.265149 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:29.266493 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:29.267104 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:29.267271 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:29.271736 (+ 4465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:29.271748 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:29.271764 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:29.271797 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:29.274686 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:29.275340 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:29.275347 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:29.275349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:29.275605 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:29.275953 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:29.275956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:29.275958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:29.276289 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:29.276514 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:29.276806 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 11:34:29.276999 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":380,"threads_started":1}
I20260924 11:34:29.278666 13081 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:34:29.396937 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 12958
I20260924 11:34:29.412767 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 12598
I20260924 11:34:29.413893 12660 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:34:29.524598 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 12598
2026-09-24T11:34:29Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4089 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:34:29 dist-test-slave-cq7c krb5kdc[13114](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:34:29 dist-test-slave-cq7c krb5kdc[13114](info): set up 2 sockets
Sep 24 11:34:29 dist-test-slave-cq7c krb5kdc[13114](info): commencing operation
krb5kdc: starting...
W20260924 11:34:30.656261 6992 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.008s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Sep 24 11:34:30 dist-test-slave-cq7c krb5kdc[13114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:34:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:34:30Z Disabled control of system clock
WARNING: no policy specified for oryx/127.6.212.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.212.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.212.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.62.keytab.
Entry for principal oryx/127.6.212.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.62.keytab.
I20260924 11:34:30.879902 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:42625
--webserver_interface=127.6.212.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.62.keytab
--principal=oryx/127.6.212.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:33661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:42625
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:34:31.204893 13127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:31.205282 13127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:31.205448 13127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:31.212666 13127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:34:31.212828 13127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:31.212937 13127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:31.213045 13127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:34:31.213143 13127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:34:31.222991 13127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:33661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:42625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.62.keytab
--principal=oryx/127.6.212.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:42625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.212.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:34:31.224442 13127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:31.225812 13127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:31.237475 13132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:31.237500 13135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:31.237406 13133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:31.238380 13127 server_base.cc:1034] running on GCE node
I20260924 11:34:31.239092 13127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:31.240662 13127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:31.241869 13127 hybrid_clock.cc:648] HybridClock initialized: now 1790249671241821 us; error 39 us; skew 500 ppm
I20260924 11:34:31.242184 13127 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:34:31 dist-test-slave-cq7c krb5kdc[13114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249671, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.212.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:31.245334 13127 init.cc:377] Logged in from keytab as oryx/127.6.212.62@KRBTEST.COM (short username oryx)
I20260924 11:34:31.246893 13127 webserver.cc:460] Webserver started at http://127.6.212.62:40511/ using document root <none> and password file <none>
I20260924 11:34:31.247679 13127 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:31.247815 13127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:31.248142 13127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:34:31.251098 13127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "f32c56b9b2b94674b1fde17e4e460006"
format_stamp: "Formatted at 2026-09-24 11:34:31 on dist-test-slave-cq7c"
server_key: "020ffdb14a2443cb0243eee8d5c70008"
server_key_iv: "7909d45ebaf21f438bf2e6d556308b76"
server_key_version: "encryptionkey@0"
I20260924 11:34:31.251955 13127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "f32c56b9b2b94674b1fde17e4e460006"
format_stamp: "Formatted at 2026-09-24 11:34:31 on dist-test-slave-cq7c"
server_key: "020ffdb14a2443cb0243eee8d5c70008"
server_key_iv: "7909d45ebaf21f438bf2e6d556308b76"
server_key_version: "encryptionkey@0"
I20260924 11:34:31.257490 13127 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 11:34:31.261230 13142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:31.262789 13127 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 11:34:31.262974 13127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "f32c56b9b2b94674b1fde17e4e460006"
format_stamp: "Formatted at 2026-09-24 11:34:31 on dist-test-slave-cq7c"
server_key: "020ffdb14a2443cb0243eee8d5c70008"
server_key_iv: "7909d45ebaf21f438bf2e6d556308b76"
server_key_version: "encryptionkey@0"
I20260924 11:34:31.263175 13127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:31.286085 13127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:31.290685 13127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:34:31.291018 13127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:31.311280 13127 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.62:42625
I20260924 11:34:31.311302 13193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.62:42625 every 8 connection(s)
I20260924 11:34:31.312794 13127 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:34:31.316799 13194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:34:31.321336 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 13127
I20260924 11:34:31.321565 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:34:31.322383 6992 external_mini_cluster.cc:1428] Setting key 2825d79b600e69e12869c4c2ffed2a22
I20260924 11:34:31.329134 13194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006: Bootstrap starting.
I20260924 11:34:31.332739 13194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006: Neither blocks nor log segments found. Creating new log.
I20260924 11:34:31.333841 13194 log.cc:824] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006: Log is configured to *not* fsync() on all Append() calls
I20260924 11:34:31.337299 13194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006: No bootstrap required, opened a new log
Sep 24 11:34:31 dist-test-slave-cq7c krb5kdc[13114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM
I20260924 11:34:31.345140 13194 raft_consensus.cc:348] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } }
I20260924 11:34:31.345522 13194 raft_consensus.cc:374] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:34:31.345657 13194 raft_consensus.cc:729] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f32c56b9b2b94674b1fde17e4e460006, State: Initialized, Role: FOLLOWER
I20260924 11:34:31.346236 13194 consensus_queue.cc:260] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } }
I20260924 11:34:31.346362 13197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:31.324713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57044 (local address 127.6.212.62:42625)
0924 11:34:31.325538 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:31.325550 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:31.325599 (+ 49us) server_negotiation.cc:407] Connection header received
0924 11:34:31.327568 (+ 1969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:31.327594 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:31.327985 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:31.328308 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:31.330058 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:31.331431 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:31.333157 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:31.333805 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:31.333930 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:31.338636 (+ 4706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:31.338643 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:31.338657 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:31.338692 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:31.341927 (+ 3235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:31.342672 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:31.342683 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:31.342698 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:31.342874 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:31.343273 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:31.343278 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:31.343283 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:31.343887 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:31.344200 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:31.344635 (+ 435us) server_negotiation.cc:299] Negotiation successful
0924 11:34:31.345069 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":406,"threads_started":1}
I20260924 11:34:31.346474 13194 raft_consensus.cc:388] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:34:31.346608 13194 raft_consensus.cc:482] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:34:31.346786 13194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:34:31.349155 13194 raft_consensus.cc:504] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } }
I20260924 11:34:31.349661 13194 leader_election.cc:302] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f32c56b9b2b94674b1fde17e4e460006; no voters:
I20260924 11:34:31.350193 13194 leader_election.cc:288] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:34:31.350322 13199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:34:31.351023 13199 raft_consensus.cc:686] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 [term 1 LEADER]: Becoming Leader. State: Replica: f32c56b9b2b94674b1fde17e4e460006, State: Running, Role: LEADER
I20260924 11:34:31.351627 13199 consensus_queue.cc:237] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } }
I20260924 11:34:31.352360 13194 sys_catalog.cc:564] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:34:31.355139 13201 sys_catalog.cc:455] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f32c56b9b2b94674b1fde17e4e460006. Latest consensus state: current_term: 1 leader_uuid: "f32c56b9b2b94674b1fde17e4e460006" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } }
I20260924 11:34:31.355119 13200 sys_catalog.cc:455] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f32c56b9b2b94674b1fde17e4e460006" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } }
I20260924 11:34:31.355599 13201 sys_catalog.cc:458] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 [sys.catalog]: This master's current role is: LEADER
I20260924 11:34:31.355607 13200 sys_catalog.cc:458] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 [sys.catalog]: This master's current role is: LEADER
I20260924 11:34:31.361093 13206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:34:31.366745 13206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:34:31.375958 13206 catalog_manager.cc:1347] Generated new cluster ID: 667ded6d01e14754a126c1667f74f666
I20260924 11:34:31.376092 13206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:34:31.402666 13206 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:34:31.404430 13206 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:34:31.414412 13206 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006: Generated new TSK 0
I20260924 11:34:31.415268 13206 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.6.212.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.212.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.212.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.1.keytab.
Entry for principal oryx/127.6.212.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.1.keytab.
I20260924 11:34:31.474943 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.1:0
--local_ip_for_outbound_sockets=127.6.212.1
--webserver_interface=127.6.212.1
--webserver_port=0
--tserver_master_addrs=127.6.212.62:42625
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.1.keytab
--principal=oryx/127.6.212.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:33661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:34:31.823849 13220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:31.824155 13220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:31.824250 13220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:31.831509 13220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:31.831630 13220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:31.831744 13220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.1
I20260924 11:34:31.841718 13220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:33661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.1.keytab
--principal=oryx/127.6.212.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.212.1
--webserver_port=0
--tserver_master_addrs=127.6.212.62:42625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.1
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:34:31.843008 13220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:31.844415 13220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:31.858098 13225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:31.858162 13228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:31.858357 13226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:31.858904 13220 server_base.cc:1034] running on GCE node
I20260924 11:34:31.859596 13220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:31.860615 13220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:31.861855 13220 hybrid_clock.cc:648] HybridClock initialized: now 1790249671861774 us; error 69 us; skew 500 ppm
I20260924 11:34:31.862214 13220 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:34:31 dist-test-slave-cq7c krb5kdc[13114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249671, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.212.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:31.865415 13220 init.cc:377] Logged in from keytab as oryx/127.6.212.1@KRBTEST.COM (short username oryx)
I20260924 11:34:31.867017 13220 webserver.cc:460] Webserver started at http://127.6.212.1:45811/ using document root <none> and password file <none>
I20260924 11:34:31.867847 13220 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:31.867974 13220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:31.868283 13220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:34:31.871178 13220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-0/data/instance:
uuid: "92c42ff207f3406aa06f3984a03fc08b"
format_stamp: "Formatted at 2026-09-24 11:34:31 on dist-test-slave-cq7c"
server_key: "0a986ebfcc227100070d3b60250e194f"
server_key_iv: "a3cd4569096dcfe3625dfad6fc8e7318"
server_key_version: "encryptionkey@0"
I20260924 11:34:31.872001 13220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance:
uuid: "92c42ff207f3406aa06f3984a03fc08b"
format_stamp: "Formatted at 2026-09-24 11:34:31 on dist-test-slave-cq7c"
server_key: "0a986ebfcc227100070d3b60250e194f"
server_key_iv: "a3cd4569096dcfe3625dfad6fc8e7318"
server_key_version: "encryptionkey@0"
I20260924 11:34:31.877146 13220 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260924 11:34:31.880548 13235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:31.881846 13220 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 11:34:31.882025 13220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-0/wal
uuid: "92c42ff207f3406aa06f3984a03fc08b"
format_stamp: "Formatted at 2026-09-24 11:34:31 on dist-test-slave-cq7c"
server_key: "0a986ebfcc227100070d3b60250e194f"
server_key_iv: "a3cd4569096dcfe3625dfad6fc8e7318"
server_key_version: "encryptionkey@0"
I20260924 11:34:31.882225 13220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:31.897393 13220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:31.901307 13220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:34:31.901631 13220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:31.902582 13220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:34:31.903925 13220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:34:31.904000 13220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:31.904165 13220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:34:31.904229 13220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:31.936636 13220 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.1:42317
I20260924 11:34:31.936723 13347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.1:42317 every 8 connection(s)
I20260924 11:34:31.938091 13220 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
I20260924 11:34:31.939119 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 13220
I20260924 11:34:31.939350 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance
I20260924 11:34:31.940140 6992 external_mini_cluster.cc:1428] Setting key 20b24495e6085b2a2d27114a0f243365
Sep 24 11:34:31 dist-test-slave-cq7c krb5kdc[13114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249671, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.212.1@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM
I20260924 11:34:31.961591 13350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:31.940949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:47655 (local address 127.6.212.62:42625)
0924 11:34:31.941377 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:31.941383 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:31.942466 (+ 1083us) server_negotiation.cc:407] Connection header received
0924 11:34:31.945244 (+ 2778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:31.945247 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:31.945309 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:31.945409 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:31.947518 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:31.948271 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:31.950594 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:31.951399 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:31.951535 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:31.955555 (+ 4020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:31.955560 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:31.955563 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:31.955592 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:31.958003 (+ 2411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:31.958925 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:31.958928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:31.958930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:31.959053 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:31.959584 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:31.959587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:31.959589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:31.959830 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:31.959997 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:31.960820 (+ 823us) server_negotiation.cc:299] Negotiation successful
0924 11:34:31.961101 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":229,"threads_started":1}
I20260924 11:34:31.962442 13351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:31.941493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.6.212.1:47655)
0924 11:34:31.942279 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:31.942335 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:31.944946 (+ 2611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:31.945695 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:31.945710 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:31.946479 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:31.947313 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:31.947342 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:31.948439 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:31.948447 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:31.950443 (+ 1996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:31.950455 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:31.951601 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:31.951606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:31.951801 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:31.952547 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:31.952586 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:31.955307 (+ 2721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:31.958154 (+ 2847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:31.958166 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:31.958186 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:31.958808 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:31.959163 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:31.959170 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:31.959176 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:31.959464 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:31.960007 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:31.960028 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:31.960421 (+ 393us) client_negotiation.cc:769] Sending connection context
0924 11:34:31.960650 (+ 229us) client_negotiation.cc:240] Negotiation successful
0924 11:34:31.961088 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":312,"threads_started":1}
WARNING: no policy specified for oryx/127.6.212.2@KRBTEST.COM; defaulting to no policy
I20260924 11:34:31.964161 13348 heartbeater.cc:347] Connected to a master server at 127.6.212.62:42625
I20260924 11:34:31.964505 13348 heartbeater.cc:464] Registering TS with master...
I20260924 11:34:31.965241 13348 heartbeater.cc:511] Master 127.6.212.62:42625 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.212.2@KRBTEST.COM" created.
I20260924 11:34:31.967595 13159 ts_manager.cc:194] Registered new tserver with Master: 92c42ff207f3406aa06f3984a03fc08b (127.6.212.1:42317)
I20260924 11:34:31.969156 13159 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.212.1@KRBTEST.COM'} at 127.6.212.1:47655
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.212.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.2.keytab.
Entry for principal oryx/127.6.212.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.2.keytab.
I20260924 11:34:31.994541 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.2:0
--local_ip_for_outbound_sockets=127.6.212.2
--webserver_interface=127.6.212.2
--webserver_port=0
--tserver_master_addrs=127.6.212.62:42625
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.2.keytab
--principal=oryx/127.6.212.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:33661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:34:32.326921 13354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:32.327211 13354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:32.327307 13354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:32.334721 13354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:32.334846 13354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:32.334968 13354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.2
I20260924 11:34:32.344920 13354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:33661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.2.keytab
--principal=oryx/127.6.212.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.212.2
--webserver_port=0
--tserver_master_addrs=127.6.212.62:42625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.2
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:34:32.346246 13354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:32.347615 13354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:32.361008 13359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:32.361047 13360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:32.361071 13362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:32.362180 13354 server_base.cc:1034] running on GCE node
I20260924 11:34:32.362907 13354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:32.363991 13354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:32.365265 13354 hybrid_clock.cc:648] HybridClock initialized: now 1790249672365197 us; error 62 us; skew 500 ppm
I20260924 11:34:32.365586 13354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:34:32 dist-test-slave-cq7c krb5kdc[13114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249672, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.212.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:32.369105 13354 init.cc:377] Logged in from keytab as oryx/127.6.212.2@KRBTEST.COM (short username oryx)
I20260924 11:34:32.370565 13354 webserver.cc:460] Webserver started at http://127.6.212.2:34395/ using document root <none> and password file <none>
I20260924 11:34:32.371244 13354 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:32.371340 13354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:32.371659 13354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:34:32.374589 13354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-1/data/instance:
uuid: "faaacac03e5747f48f17a2b84561a5d8"
format_stamp: "Formatted at 2026-09-24 11:34:32 on dist-test-slave-cq7c"
server_key: "06327aa6db175fea2a4472d47c692551"
server_key_iv: "a107241b67b5984fb7cf64bf51abe268"
server_key_version: "encryptionkey@0"
I20260924 11:34:32.375314 13354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance:
uuid: "faaacac03e5747f48f17a2b84561a5d8"
format_stamp: "Formatted at 2026-09-24 11:34:32 on dist-test-slave-cq7c"
server_key: "06327aa6db175fea2a4472d47c692551"
server_key_iv: "a107241b67b5984fb7cf64bf51abe268"
server_key_version: "encryptionkey@0"
I20260924 11:34:32.380761 13354 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 11:34:32.384364 13369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:32.385952 13354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 11:34:32.386093 13354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-1/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-1/wal
uuid: "faaacac03e5747f48f17a2b84561a5d8"
format_stamp: "Formatted at 2026-09-24 11:34:32 on dist-test-slave-cq7c"
server_key: "06327aa6db175fea2a4472d47c692551"
server_key_iv: "a107241b67b5984fb7cf64bf51abe268"
server_key_version: "encryptionkey@0"
I20260924 11:34:32.386296 13354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:32.399840 13354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:32.404421 13354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:34:32.404680 13354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:32.405581 13354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:34:32.406867 13354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:34:32.406945 13354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:32.407025 13354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:34:32.407078 13354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:32.440245 13354 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.2:34887
I20260924 11:34:32.440294 13481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.2:34887 every 8 connection(s)
I20260924 11:34:32.441534 13354 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
I20260924 11:34:32.446797 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 13354
I20260924 11:34:32.446995 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance
I20260924 11:34:32.447525 6992 external_mini_cluster.cc:1428] Setting key 2c18508cf13d75c0006e58fe56430f7b
Sep 24 11:34:32 dist-test-slave-cq7c krb5kdc[13114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249672, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.212.2@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM
I20260924 11:34:32.462833 13350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:32.444162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:35201 (local address 127.6.212.62:42625)
0924 11:34:32.444335 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:32.444339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:32.445482 (+ 1143us) server_negotiation.cc:407] Connection header received
0924 11:34:32.447515 (+ 2033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:32.447520 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:32.447642 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:32.447765 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:32.449770 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:32.450472 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:32.452119 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:32.452662 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:32.452807 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:32.456489 (+ 3682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:32.456494 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:32.456497 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:32.456528 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:32.459173 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:32.460208 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:32.460212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:32.460215 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:32.460343 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:32.460980 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:32.460984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:32.460986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:32.461317 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:32.461441 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:32.462376 (+ 935us) server_negotiation.cc:299] Negotiation successful
0924 11:34:32.462565 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 11:34:32.464500 13484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:32.444596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.6.212.2:35201)
0924 11:34:32.445312 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:32.445353 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:32.447203 (+ 1850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:32.448041 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:32.448052 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:32.448775 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:32.449608 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:32.449636 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:32.450633 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:32.450638 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:32.451971 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:32.451982 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:32.452940 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:32.452950 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:32.453220 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:32.453987 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:32.454009 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:32.456260 (+ 2251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:32.459330 (+ 3070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:32.459343 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:32.459398 (+ 55us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:32.460064 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:32.460507 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:32.460514 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:32.460520 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:32.460819 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:32.461448 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:32.461459 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:32.461945 (+ 486us) client_negotiation.cc:769] Sending connection context
0924 11:34:32.462289 (+ 344us) client_negotiation.cc:240] Negotiation successful
0924 11:34:32.462906 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":286,"threads_started":1}
I20260924 11:34:32.466403 13482 heartbeater.cc:347] Connected to a master server at 127.6.212.62:42625
I20260924 11:34:32.466786 13482 heartbeater.cc:464] Registering TS with master...
I20260924 11:34:32.467736 13482 heartbeater.cc:511] Master 127.6.212.62:42625 requested a full tablet report, sending...
I20260924 11:34:32.469391 13159 ts_manager.cc:194] Registered new tserver with Master: faaacac03e5747f48f17a2b84561a5d8 (127.6.212.2:34887)
WARNING: no policy specified for oryx/127.6.212.3@KRBTEST.COM; defaulting to no policy
I20260924 11:34:32.470364 13159 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.212.2@KRBTEST.COM'} at 127.6.212.2:35201
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.212.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.212.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.3.keytab.
Entry for principal oryx/127.6.212.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.3.keytab.
I20260924 11:34:32.499310 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.3:0
--local_ip_for_outbound_sockets=127.6.212.3
--webserver_interface=127.6.212.3
--webserver_port=0
--tserver_master_addrs=127.6.212.62:42625
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.3.keytab
--principal=oryx/127.6.212.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:33661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:34:32.824688 13487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:32.825057 13487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:32.825230 13487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:32.832690 13487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:32.832865 13487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:32.833055 13487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.3
I20260924 11:34:32.843132 13487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:33661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.3.keytab
--principal=oryx/127.6.212.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.212.3
--webserver_port=0
--tserver_master_addrs=127.6.212.62:42625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.3
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:34:32.844605 13487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:32.846012 13487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:32.859419 13495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:32.859495 13492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:32.859534 13493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:32.860363 13487 server_base.cc:1034] running on GCE node
I20260924 11:34:32.861070 13487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:32.862180 13487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:32.863440 13487 hybrid_clock.cc:648] HybridClock initialized: now 1790249672863375 us; error 56 us; skew 500 ppm
I20260924 11:34:32.863807 13487 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:34:32 dist-test-slave-cq7c krb5kdc[13114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249672, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.212.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:32.867201 13487 init.cc:377] Logged in from keytab as oryx/127.6.212.3@KRBTEST.COM (short username oryx)
I20260924 11:34:32.868908 13487 webserver.cc:460] Webserver started at http://127.6.212.3:45459/ using document root <none> and password file <none>
I20260924 11:34:32.869638 13487 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:32.869782 13487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:32.870086 13487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:34:32.873018 13487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-2/data/instance:
uuid: "b1b300a0baf545cebef745d22579f93e"
format_stamp: "Formatted at 2026-09-24 11:34:32 on dist-test-slave-cq7c"
server_key: "04b674aa114a84e423fc153658b05d11"
server_key_iv: "e391d6924008b9ec08bc36a7b6e01195"
server_key_version: "encryptionkey@0"
I20260924 11:34:32.873780 13487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance:
uuid: "b1b300a0baf545cebef745d22579f93e"
format_stamp: "Formatted at 2026-09-24 11:34:32 on dist-test-slave-cq7c"
server_key: "04b674aa114a84e423fc153658b05d11"
server_key_iv: "e391d6924008b9ec08bc36a7b6e01195"
server_key_version: "encryptionkey@0"
I20260924 11:34:32.878953 13487 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260924 11:34:32.882515 13502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:32.884099 13487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:34:32.884274 13487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-2/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-2/wal
uuid: "b1b300a0baf545cebef745d22579f93e"
format_stamp: "Formatted at 2026-09-24 11:34:32 on dist-test-slave-cq7c"
server_key: "04b674aa114a84e423fc153658b05d11"
server_key_iv: "e391d6924008b9ec08bc36a7b6e01195"
server_key_version: "encryptionkey@0"
I20260924 11:34:32.884466 13487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:32.897441 13487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:32.901968 13487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:34:32.902266 13487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:32.903264 13487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:34:32.904631 13487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:34:32.904735 13487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:32.904877 13487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:34:32.904949 13487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:32.938024 13487 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.3:45033
I20260924 11:34:32.938067 13614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.3:45033 every 8 connection(s)
I20260924 11:34:32.939296 13487 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
I20260924 11:34:32.939774 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 13487
I20260924 11:34:32.940038 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance
I20260924 11:34:32.940694 6992 external_mini_cluster.cc:1428] Setting key 2e9c5e803b60aece09d63f1c729a773b
Sep 24 11:34:32 dist-test-slave-cq7c krb5kdc[13114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249672, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.212.3@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM
I20260924 11:34:32.961423 13350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:32.942327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:37045 (local address 127.6.212.62:42625)
0924 11:34:32.942521 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:32.942527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:32.944067 (+ 1540us) server_negotiation.cc:407] Connection header received
0924 11:34:32.945887 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:32.945891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:32.945956 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:32.946045 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:32.947531 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:32.948262 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:32.950141 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:32.950566 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:32.950670 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:32.954716 (+ 4046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:32.954724 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:32.954729 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:32.954772 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:32.957951 (+ 3179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:32.958761 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:32.958775 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:32.958779 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:32.958910 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:32.959407 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:32.959415 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:32.959426 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:32.959745 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:32.959915 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:32.960780 (+ 865us) server_negotiation.cc:299] Negotiation successful
0924 11:34:32.961040 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 11:34:32.962594 13617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:32.942973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.6.212.3:37045)
0924 11:34:32.943923 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:32.943968 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:32.945643 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:32.946223 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:32.946231 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:32.946755 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:32.947322 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:32.947339 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:32.948431 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:32.948436 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:32.949844 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:32.949857 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:32.950786 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:32.950792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:32.951027 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:32.951880 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:32.951905 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:32.954470 (+ 2565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:32.958114 (+ 3644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:32.958124 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:32.958145 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:32.958623 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:32.959033 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:32.959038 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:32.959043 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:32.959245 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:32.959886 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:32.959895 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:32.960275 (+ 380us) client_negotiation.cc:769] Sending connection context
0924 11:34:32.960657 (+ 382us) client_negotiation.cc:240] Negotiation successful
0924 11:34:32.961251 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":396,"threads_started":1}
I20260924 11:34:32.964370 13615 heartbeater.cc:347] Connected to a master server at 127.6.212.62:42625
I20260924 11:34:32.964785 13615 heartbeater.cc:464] Registering TS with master...
I20260924 11:34:32.965544 13615 heartbeater.cc:511] Master 127.6.212.62:42625 requested a full tablet report, sending...
I20260924 11:34:32.967072 13159 ts_manager.cc:194] Registered new tserver with Master: b1b300a0baf545cebef745d22579f93e (127.6.212.3:45033)
I20260924 11:34:32.968111 13159 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.212.3@KRBTEST.COM'} at 127.6.212.3:37045
I20260924 11:34:32.969497 6992 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:34:32.972671 13348 heartbeater.cc:503] Master 127.6.212.62:42625 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.6.212.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.212.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.212.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.61.keytab.
Entry for principal oryx/127.6.212.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.61.keytab.
I20260924 11:34:33.019212 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.61:38525
--webserver_interface=127.6.212.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.61.keytab
--principal=oryx/127.6.212.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:33661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:42625,127.6.212.61:38525
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:34:33.378849 13620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:33.379184 13620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:33.379405 13620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:33.387183 13620 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:34:33.387311 13620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:33.387413 13620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:33.387521 13620 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:34:33.387583 13620 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:34:33.398634 13620 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:33661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:42625,127.6.212.61:38525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.61.keytab
--principal=oryx/127.6.212.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.61:38525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.212.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
W20260924 11:34:33.399104 13620 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.6.212.62:42625,127.6.212.61:38525'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 11:34:33.400205 13620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:33.401475 13620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:33.413564 13625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:33.413527 13628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:33.413614 13626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:33.416815 13620 server_base.cc:1034] running on GCE node
I20260924 11:34:33.417635 13620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:33.419092 13620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:33.420387 13620 hybrid_clock.cc:648] HybridClock initialized: now 1790249673420275 us; error 45 us; skew 500 ppm
I20260924 11:34:33.420761 13620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:34:33 dist-test-slave-cq7c krb5kdc[13114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249673, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.212.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:33.424026 13620 init.cc:377] Logged in from keytab as oryx/127.6.212.61@KRBTEST.COM (short username oryx)
I20260924 11:34:33.425755 13620 webserver.cc:460] Webserver started at http://127.6.212.61:36721/ using document root <none> and password file <none>
I20260924 11:34:33.426501 13620 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:33.426635 13620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:33.426959 13620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:34:33.430017 13620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-1/data/instance:
uuid: "d211d01c60f2417bb346495d00f5593f"
format_stamp: "Formatted at 2026-09-24 11:34:33 on dist-test-slave-cq7c"
server_key: "1e3e32da54fdabc1834a8b782af07121"
server_key_iv: "83d4252e97bbfce504706338590587fc"
server_key_version: "encryptionkey@0"
I20260924 11:34:33.430800 13620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-1/wal/instance:
uuid: "d211d01c60f2417bb346495d00f5593f"
format_stamp: "Formatted at 2026-09-24 11:34:33 on dist-test-slave-cq7c"
server_key: "1e3e32da54fdabc1834a8b782af07121"
server_key_iv: "83d4252e97bbfce504706338590587fc"
server_key_version: "encryptionkey@0"
I20260924 11:34:33.436124 13620 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 11:34:33.439752 13635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:33.441214 13620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:34:33.441390 13620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-1/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-1/wal
uuid: "d211d01c60f2417bb346495d00f5593f"
format_stamp: "Formatted at 2026-09-24 11:34:33 on dist-test-slave-cq7c"
server_key: "1e3e32da54fdabc1834a8b782af07121"
server_key_iv: "83d4252e97bbfce504706338590587fc"
server_key_version: "encryptionkey@0"
I20260924 11:34:33.441653 13620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:33.462600 13620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:33.467530 13620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:34:33.467818 13620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:33.473217 13482 heartbeater.cc:503] Master 127.6.212.62:42625 was elected leader, sending a full tablet report...
I20260924 11:34:33.487247 13620 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.61:38525
I20260924 11:34:33.487267 13686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.61:38525 every 8 connection(s)
I20260924 11:34:33.488776 13620 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-1/data/info.pb
I20260924 11:34:33.490684 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 13620
I20260924 11:34:33.493167 13687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:34:33.499979 13687 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:34:33 dist-test-slave-cq7c krb5kdc[13114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.212.61@KRBTEST.COM
Sep 24 11:34:33 dist-test-slave-cq7c krb5kdc[13114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249673, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.212.61@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM
I20260924 11:34:33.514840 13690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:33.492836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38956 (local address 127.6.212.61:38525)
0924 11:34:33.493615 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:33.493627 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:33.493666 (+ 39us) server_negotiation.cc:407] Connection header received
0924 11:34:33.495234 (+ 1568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:33.495253 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:33.495729 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:33.496134 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:33.497625 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:33.498904 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:33.500192 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:33.500928 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:33.501135 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:33.506103 (+ 4968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:33.506111 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:33.506130 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:33.506168 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:33.510006 (+ 3838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:33.510715 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:33.510725 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:33.510733 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:33.511019 (+ 286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:33.511404 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:33.511416 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:33.511419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:33.512042 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:33.512367 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:33.512790 (+ 423us) server_negotiation.cc:299] Negotiation successful
0924 11:34:33.513272 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":347,"threads_started":1}
I20260924 11:34:33.518850 13691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:33.502542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.0.0.1:57046)
0924 11:34:33.503520 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:33.503584 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:33.503762 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:33.504516 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:33.504526 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:33.505199 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:33.505449 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:33.505457 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:33.506682 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:33.506690 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:33.508622 (+ 1932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:33.508638 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:33.509355 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:33.509364 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:33.509595 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:33.510312 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:33.510332 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:33.512932 (+ 2600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:33.516429 (+ 3497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:33.516442 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:33.516449 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:33.516859 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:33.517241 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:33.517246 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:33.517250 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:33.517387 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:33.518049 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:33.518054 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:33.518215 (+ 161us) client_negotiation.cc:769] Sending connection context
0924 11:34:33.518359 (+ 144us) client_negotiation.cc:240] Negotiation successful
0924 11:34:33.518580 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":467,"threads_started":1}
I20260924 11:34:33.518977 13692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:33.502596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57046 (local address 127.6.212.62:42625)
0924 11:34:33.504196 (+ 1600us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:33.504200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:33.504215 (+ 15us) server_negotiation.cc:407] Connection header received
0924 11:34:33.504260 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:33.504264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:33.504323 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:33.504410 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:33.505610 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:33.506506 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:33.508766 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:33.509149 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:33.509243 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:33.513143 (+ 3900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:33.513151 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:33.513156 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:33.513203 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:33.516295 (+ 3092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:33.516993 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:33.517003 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:33.517006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:33.517127 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:33.517550 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:33.517555 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:33.517558 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:33.517915 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:33.518057 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:33.518508 (+ 451us) server_negotiation.cc:299] Negotiation successful
0924 11:34:33.518677 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1507,"thread_start_us":1412,"threads_started":1}
I20260924 11:34:33.520499 13687 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.212.61" port: 38525 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:34:33 dist-test-slave-cq7c krb5kdc[13114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249673, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.212.61@KRBTEST.COM for oryx/127.6.212.61@KRBTEST.COM
I20260924 11:34:33.532980 13691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:33.521263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:38525 (local address 127.0.0.1:38960)
0924 11:34:33.521476 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:33.521497 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:33.521654 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:33.521956 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:33.521961 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:33.522321 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:33.522460 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:33.522468 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:33.523542 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:33.523548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:33.524424 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:33.524435 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:33.525139 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:33.525145 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:33.525391 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:33.525974 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:33.525991 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:33.527964 (+ 1973us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:33.530628 (+ 2664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:33.530635 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:33.530640 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:33.530985 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:33.531352 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:33.531380 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:33.531383 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:33.531518 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:33.531990 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:33.531994 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:33.532140 (+ 146us) client_negotiation.cc:769] Sending connection context
0924 11:34:33.532274 (+ 134us) client_negotiation.cc:240] Negotiation successful
0924 11:34:33.532567 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"mutex_wait_us":41}
I20260924 11:34:33.533049 13690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:33.521292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38960 (local address 127.6.212.61:38525)
0924 11:34:33.521473 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:33.521479 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:33.521577 (+ 98us) server_negotiation.cc:407] Connection header received
0924 11:34:33.521769 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:33.521776 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:33.521839 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:33.521943 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:33.522595 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:33.523341 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:33.524564 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:33.524992 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:33.525133 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:33.528128 (+ 2995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:33.528133 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:33.528138 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:33.528175 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:33.530482 (+ 2307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:33.531097 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:33.531105 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:33.531109 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:33.531241 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:33.531621 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:33.531629 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:33.531632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:33.531878 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:33.532031 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:33.532351 (+ 320us) server_negotiation.cc:299] Negotiation successful
0924 11:34:33.532700 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":156,"server-negotiator.queue_time_us":57}
I20260924 11:34:33.537837 13687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f: Bootstrap starting.
I20260924 11:34:33.540990 13687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f: Neither blocks nor log segments found. Creating new log.
I20260924 11:34:33.542085 13687 log.cc:824] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f: Log is configured to *not* fsync() on all Append() calls
I20260924 11:34:33.544788 13687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f: No bootstrap required, opened a new log
I20260924 11:34:33.551816 13687 raft_consensus.cc:348] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } peers { permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 38525 } }
I20260924 11:34:33.552186 13687 raft_consensus.cc:374] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:34:33.552279 13687 raft_consensus.cc:729] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d211d01c60f2417bb346495d00f5593f, State: Initialized, Role: FOLLOWER
I20260924 11:34:33.552861 13687 consensus_queue.cc:260] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } peers { permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 38525 } }
I20260924 11:34:33.553668 13694 sys_catalog.cc:455] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } peers { permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 38525 } } }
I20260924 11:34:33.554065 13694 sys_catalog.cc:458] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:34:33.554692 13687 sys_catalog.cc:564] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [sys.catalog]: configured and running, proceeding with master startup.
W20260924 11:34:33.563264 13705 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 11:34:33.563429 13705 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 11:34:33.565148 13620 master_runner.cc:428] Detected that this master d211d01c60f2417bb346495d00f5593f is joining an existing cluster
I20260924 11:34:33.565253 13620 master_runner.cc:432] Initiating AddMaster RPC to add 127.6.212.61:38525
I20260924 11:34:33.595052 13692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:33.570180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57062 (local address 127.6.212.62:42625)
0924 11:34:33.570587 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:33.570592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:33.578345 (+ 7753us) server_negotiation.cc:407] Connection header received
0924 11:34:33.578891 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:33.578896 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:33.579010 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:33.579115 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:33.580499 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:33.581556 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:33.583938 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:33.584613 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:33.584711 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:33.587892 (+ 3181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:33.587897 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:33.587902 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:33.587936 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:33.592117 (+ 4181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:33.592876 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:33.592882 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:33.592885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:33.593052 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:33.593449 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:33.593457 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:33.593461 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:33.593787 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:33.593904 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:33.594506 (+ 602us) server_negotiation.cc:299] Negotiation successful
0924 11:34:33.594708 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273}
I20260924 11:34:33.595052 13690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:33.570146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38962 (local address 127.6.212.61:38525)
0924 11:34:33.570419 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:33.570430 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:33.578001 (+ 7571us) server_negotiation.cc:407] Connection header received
0924 11:34:33.578503 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:33.578511 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:33.578583 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:33.578688 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:33.580164 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:33.581819 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:33.583923 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:33.584625 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:33.584711 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:33.587878 (+ 3167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:33.587884 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:33.587888 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:33.587918 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:33.591760 (+ 3842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:33.592614 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:33.592621 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:33.592626 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:33.592757 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:33.593367 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:33.593374 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:33.593377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:33.593751 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:33.593910 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:33.594458 (+ 548us) server_negotiation.cc:299] Negotiation successful
0924 11:34:33.594685 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
WARNING: no policy specified for oryx/127.6.212.60@KRBTEST.COM; defaulting to no policy
I20260924 11:34:33.596222 13711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:33.570906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.0.0.1:57062)
0924 11:34:33.578213 (+ 7307us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:33.578233 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:33.578371 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:33.579290 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:33.579296 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:33.580019 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:33.580346 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:33.580356 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:33.581780 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:33.581786 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:33.583778 (+ 1992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:33.583792 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:33.585013 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:33.585020 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:33.585325 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:33.586330 (+ 1005us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:33.586350 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:33.587681 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:33.592280 (+ 4599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:33.592284 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:33.592287 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:33.592751 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:33.593174 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:33.593178 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:33.593181 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:33.593332 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:33.593973 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:33.593988 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:33.594237 (+ 249us) client_negotiation.cc:769] Sending connection context
0924 11:34:33.595333 (+ 1096us) client_negotiation.cc:240] Negotiation successful
0924 11:34:33.595731 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7152,"mutex_wait_us":243,"spinlock_wait_cycles":44800,"thread_start_us":7077,"threads_started":1}
I20260924 11:34:33.596524 13712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:33.570488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:38525 (local address 127.0.0.1:38962)
0924 11:34:33.577855 (+ 7367us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:33.577887 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:33.578334 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:33.578712 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:33.578716 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:33.579577 (+ 861us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:33.580000 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:33.580013 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:33.582054 (+ 2041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:33.582060 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:33.583778 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:33.583792 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:33.585021 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:33.585026 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:33.585324 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:33.586249 (+ 925us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:33.586279 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:33.587724 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:33.591915 (+ 4191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:33.591924 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:33.591929 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:33.592500 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:33.592873 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:33.592877 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:33.592880 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:33.593264 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:33.593973 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:33.593977 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:33.594204 (+ 227us) client_negotiation.cc:769] Sending connection context
0924 11:34:33.595715 (+ 1511us) client_negotiation.cc:240] Negotiation successful
0924 11:34:33.595993 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7138,"mutex_wait_us":216,"spinlock_wait_cycles":29824,"thread_start_us":7405,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.212.60@KRBTEST.COM" created.
Sep 24 11:34:33 dist-test-slave-cq7c krb5kdc[13114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249671, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.212.62@KRBTEST.COM for oryx/127.6.212.61@KRBTEST.COM
I20260924 11:34:33.623423 13715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:33.608630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:38525 (local address 127.0.0.1:38976)
0924 11:34:33.609330 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:33.609365 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:33.609593 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:33.610047 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:33.610055 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:33.610683 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:33.610888 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:33.610897 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:33.612016 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:33.612022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:33.612998 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:33.613013 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:33.613784 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:33.613790 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:33.613988 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:33.614625 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:33.614643 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:33.617056 (+ 2413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:33.620290 (+ 3234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:33.620304 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:33.620312 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:33.620951 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:33.621385 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:33.621387 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:33.621390 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:33.621517 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:33.622275 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:33.622281 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:33.622554 (+ 273us) client_negotiation.cc:769] Sending connection context
0924 11:34:33.622731 (+ 177us) client_negotiation.cc:240] Negotiation successful
0924 11:34:33.622974 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":342,"threads_started":1}
I20260924 11:34:33.623847 13690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:33.608629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38976 (local address 127.6.212.61:38525)
0924 11:34:33.608822 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:33.608831 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:33.609453 (+ 622us) server_negotiation.cc:407] Connection header received
0924 11:34:33.609804 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:33.609811 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:33.609882 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:33.609988 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:33.611050 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:33.611881 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:33.613140 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:33.613605 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:33.613768 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:33.617242 (+ 3474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:33.617247 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:33.617250 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:33.617278 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:33.620115 (+ 2837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:33.621093 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:33.621097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:33.621100 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:33.621259 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:33.621660 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:33.621668 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:33.621674 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:33.622123 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:33.622288 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:33.622870 (+ 582us) server_negotiation.cc:299] Negotiation successful
0924 11:34:33.623109 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 11:34:33.625720 13159 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.6.212.61:38525: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d211d01c60f2417bb346495d00f5593f"
member_type: NON_VOTER
last_known_addr {
host: "127.6.212.61"
port: 38525
}
attrs {
promote: true
}
}
dest_uuid: "f32c56b9b2b94674b1fde17e4e460006"
cas_config_opid_index: -1
I20260924 11:34:33.626783 13159 consensus_queue.cc:237] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } peers { permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: NON_VOTER last_known_addr { host: "127.6.212.61" port: 38525 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.212.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.60.keytab.
Entry for principal oryx/127.6.212.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.60.keytab.
I20260924 11:34:33.630352 13717 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER d211d01c60f2417bb346495d00f5593f (127.6.212.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } peers { permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: NON_VOTER last_known_addr { host: "127.6.212.61" port: 38525 } attrs { promote: true } } }
I20260924 11:34:33.631382 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.60:43695
--webserver_interface=127.6.212.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.60.keytab
--principal=oryx/127.6.212.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:33661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:42625,127.6.212.61:38525,127.6.212.60:43695
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:34:33.634616 13717 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.6.212.61:38525
I20260924 11:34:33.635321 13620 master.cc:537] Master@127.6.212.61:38525 shutting down...
I20260924 11:34:33.636075 13717 sys_catalog.cc:455] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f32c56b9b2b94674b1fde17e4e460006" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } peers { permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: NON_VOTER last_known_addr { host: "127.6.212.61" port: 38525 } attrs { promote: true } } }
I20260924 11:34:33.636837 13717 sys_catalog.cc:458] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 [sys.catalog]: This master's current role is: LEADER
I20260924 11:34:33.636916 13662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:34:33.645449 13662 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [term 1 FOLLOWER]: Refusing update from remote peer f32c56b9b2b94674b1fde17e4e460006: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 11:34:33.646716 13717 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: NON_VOTER last_known_addr { host: "127.6.212.61" port: 38525 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 11:34:33.649781 13146 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260924 11:34:33.650082 13620 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:34:33.650470 13620 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:34:33.650662 13620 tablet_replica.cc:330] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f: stopping tablet replica
I20260924 11:34:33.658636 13620 master.cc:552] Master@127.6.212.61:38525 shutdown complete.
I20260924 11:34:33.658727 13620 master_runner.cc:305] Clearing existing system tablet
I20260924 11:34:33.659194 13620 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:33.659305 13620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:33.663764 13715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:33.663309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:38525 (local address 127.0.0.1:38992)
0924 11:34:33.663499 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:33.663621 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.61:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 11:34:33.664080 13146 consensus_peers.cc:487] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 -> Peer d211d01c60f2417bb346495d00f5593f (127.6.212.61:38525): Couldn't send request to peer d211d01c60f2417bb346495d00f5593f. Status: Network error: Client connection negotiation failed: client connection to 127.6.212.61:38525: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 11:34:33.672075 13620 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.004s sys 0.006s
I20260924 11:34:33.673725 13723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:33.674104 13620 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:33.674357 13620 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 11:34:33.674448 13620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-1/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-1/wal
uuid: "d211d01c60f2417bb346495d00f5593f"
format_stamp: "Formatted at 2026-09-24 11:34:33 on dist-test-slave-cq7c"
server_key: "1e3e32da54fdabc1834a8b782af07121"
server_key_iv: "83d4252e97bbfce504706338590587fc"
server_key_version: "encryptionkey@0"
I20260924 11:34:33.679663 13620 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:34:33.686426 13620 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 11:34:33.686635 13620 log.cc:1196] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 11:34:33.687040 13620 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f: Deleting consensus metadata
I20260924 11:34:33.687285 13620 master_runner.cc:315] Copying system tablet from 127.6.212.62:42625
I20260924 11:34:33.692974 13620 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f: tablet copy: Beginning tablet copy session from remote peer at address 127.6.212.62:42625
I20260924 11:34:33.714323 13692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:33.694103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57070 (local address 127.6.212.62:42625)
0924 11:34:33.694270 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:33.694276 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:33.694374 (+ 98us) server_negotiation.cc:407] Connection header received
0924 11:34:33.694686 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:33.694691 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:33.694775 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:33.694882 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:33.695891 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:33.697015 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:33.698859 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:33.699575 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:33.699700 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:33.702081 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:33.702088 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:33.702093 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:33.702133 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:33.710326 (+ 8193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:33.711971 (+ 1645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:33.711986 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:33.711990 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:33.712147 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:33.712690 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:33.712695 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:33.712701 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:33.713074 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:33.713226 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:33.713707 (+ 481us) server_negotiation.cc:299] Negotiation successful
0924 11:34:33.713934 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 11:34:33.714353 13729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:33.693740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.0.0.1:57070)
0924 11:34:33.694256 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:33.694281 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:33.694504 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:33.694930 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:33.694935 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:33.695462 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:33.695733 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:33.695744 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:33.697186 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:33.697192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:33.698688 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:33.698704 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:33.699761 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:33.699767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:33.699952 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:33.700694 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:33.700718 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:33.701803 (+ 1085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:33.710706 (+ 8903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:33.710715 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:33.710721 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:33.711780 (+ 1059us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:33.712305 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:33.712312 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:33.712319 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:33.712550 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:33.713212 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:33.713218 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:33.713395 (+ 177us) client_negotiation.cc:769] Sending connection context
0924 11:34:33.713590 (+ 195us) client_negotiation.cc:240] Negotiation successful
0924 11:34:33.713910 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":541,"threads_started":1}
I20260924 11:34:33.715588 13179 tablet_copy_service.cc:140] P f32c56b9b2b94674b1fde17e4e460006: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d211d01c60f2417bb346495d00f5593f ({username='oryx', principal='oryx/127.6.212.61@KRBTEST.COM'} at 127.0.0.1:57070)
I20260924 11:34:33.715858 13179 tablet_copy_service.cc:161] P f32c56b9b2b94674b1fde17e4e460006: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d211d01c60f2417bb346495d00f5593f at {username='oryx', principal='oryx/127.6.212.61@KRBTEST.COM'} at 127.0.0.1:57070: session id = d211d01c60f2417bb346495d00f5593f-00000000000000000000000000000000
I20260924 11:34:33.720149 13179 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 11:34:33.722455 13620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 11:34:33.729429 13620 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f: tablet copy: Starting download of 0 data blocks...
I20260924 11:34:33.729884 13620 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f: tablet copy: Starting download of 1 WAL segments...
I20260924 11:34:33.736835 13620 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 11:34:33.741641 13179 tablet_copy_service.cc:342] P f32c56b9b2b94674b1fde17e4e460006: Request end of tablet copy session d211d01c60f2417bb346495d00f5593f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.6.212.61@KRBTEST.COM'} at 127.0.0.1:57070
I20260924 11:34:33.741953 13179 tablet_copy_service.cc:433] P f32c56b9b2b94674b1fde17e4e460006: ending tablet copy session d211d01c60f2417bb346495d00f5593f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d211d01c60f2417bb346495d00f5593f
I20260924 11:34:33.759063 13620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:33.784271 13730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:33.788030 13731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:33.797091 13733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:33.802934 13620 server_base.cc:1034] running on GCE node
I20260924 11:34:33.803553 13620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:33.815872 13620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:33.817162 13620 hybrid_clock.cc:648] HybridClock initialized: now 1790249673817097 us; error 45 us; skew 500 ppm
I20260924 11:34:33.817461 13620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:34:33 dist-test-slave-cq7c krb5kdc[13114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249673, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.212.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:33.819805 13620 init.cc:377] Logged in from keytab as oryx/127.6.212.61@KRBTEST.COM (short username oryx)
I20260924 11:34:33.841220 13620 webserver.cc:460] Webserver started at http://127.6.212.61:37489/ using document root <none> and password file <none>
I20260924 11:34:33.841655 13620 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:33.841760 13620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:33.850960 13620 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.004s sys 0.005s
I20260924 11:34:33.861694 13740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:33.862197 13620 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.002s sys 0.007s
I20260924 11:34:33.862367 13620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-1/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-1/wal
uuid: "d211d01c60f2417bb346495d00f5593f"
format_stamp: "Formatted at 2026-09-24 11:34:33 on dist-test-slave-cq7c"
server_key: "1e3e32da54fdabc1834a8b782af07121"
server_key_iv: "83d4252e97bbfce504706338590587fc"
server_key_version: "encryptionkey@0"
I20260924 11:34:33.862552 13620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:33.920360 13620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:33.924782 13620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:33.971040 13615 heartbeater.cc:503] Master 127.6.212.62:42625 was elected leader, sending a full tablet report...
I20260924 11:34:34.064560 13715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:34.063981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:38525 (local address 127.0.0.1:39000)
0924 11:34:34.064176 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:34.064304 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.61:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 11:34:34.066071 13715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:34.065450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:38525 (local address 127.0.0.1:39012)
0924 11:34:34.065761 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:34.065882 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.61:38525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173}
I20260924 11:34:34.099581 13620 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.61:38525
I20260924 11:34:34.099610 13791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.61:38525 every 8 connection(s)
I20260924 11:34:34.100930 13620 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-1/data/info.pb
I20260924 11:34:34.106844 13792 sys_catalog.cc:263] Verifying existing consensus state
I20260924 11:34:34.111501 13792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f: Bootstrap starting.
I20260924 11:34:34.138232 13792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 11:34:34.138878 13792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f: Bootstrap complete.
I20260924 11:34:34.140157 13792 raft_consensus.cc:348] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } peers { permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: NON_VOTER last_known_addr { host: "127.6.212.61" port: 38525 } attrs { promote: true } }
I20260924 11:34:34.140532 13792 raft_consensus.cc:729] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d211d01c60f2417bb346495d00f5593f, State: Initialized, Role: LEARNER
I20260924 11:34:34.140803 13792 consensus_queue.cc:260] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } peers { permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: NON_VOTER last_known_addr { host: "127.6.212.61" port: 38525 } attrs { promote: true } }
I20260924 11:34:34.142560 13795 sys_catalog.cc:455] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } peers { permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: NON_VOTER last_known_addr { host: "127.6.212.61" port: 38525 } attrs { promote: true } } }
I20260924 11:34:34.142915 13795 sys_catalog.cc:458] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [sys.catalog]: This master's current role is: LEARNER
I20260924 11:34:34.143244 13792 sys_catalog.cc:564] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:34:34.154873 13806 catalog_manager.cc:1259] Loaded cluster ID: 667ded6d01e14754a126c1667f74f666
I20260924 11:34:34.155025 13806 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f: loading cluster ID for follower catalog manager: success
I20260924 11:34:34.160029 13806 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f: acquiring CA information for follower catalog manager: success
I20260924 11:34:34.163017 13806 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 11:34:34.186367 13720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:34.186683 13720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:34.186779 13720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:34.195597 13720 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:34:34.195727 13720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:34.195778 13720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:34.195850 13720 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:34:34.195901 13720 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:34:34.206280 13720 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:33661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:42625,127.6.212.61:38525,127.6.212.60:43695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.60.keytab
--principal=oryx/127.6.212.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.60:43695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.212.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:34:34.207762 13720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:34.209093 13720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:34.221629 13811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:34.221580 13814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:34.221827 13812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:34.222965 13720 server_base.cc:1034] running on GCE node
I20260924 11:34:34.223903 13720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:34.225459 13720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:34.226688 13720 hybrid_clock.cc:648] HybridClock initialized: now 1790249674226634 us; error 50 us; skew 500 ppm
I20260924 11:34:34.227049 13720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:34:34 dist-test-slave-cq7c krb5kdc[13114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249674, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.212.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:34.230612 13720 init.cc:377] Logged in from keytab as oryx/127.6.212.60@KRBTEST.COM (short username oryx)
I20260924 11:34:34.232800 13720 webserver.cc:460] Webserver started at http://127.6.212.60:46539/ using document root <none> and password file <none>
I20260924 11:34:34.233537 13720 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:34.233651 13720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:34.233925 13720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:34:34.237309 13720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-2/data/instance:
uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5"
format_stamp: "Formatted at 2026-09-24 11:34:34 on dist-test-slave-cq7c"
server_key: "81ad0f7f583314e612bf85f2d16c9a31"
server_key_iv: "87b892368ee0ff00b02cf4d61af490a3"
server_key_version: "encryptionkey@0"
I20260924 11:34:34.238271 13720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-2/wal/instance:
uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5"
format_stamp: "Formatted at 2026-09-24 11:34:34 on dist-test-slave-cq7c"
server_key: "81ad0f7f583314e612bf85f2d16c9a31"
server_key_iv: "87b892368ee0ff00b02cf4d61af490a3"
server_key_version: "encryptionkey@0"
I20260924 11:34:34.244431 13720 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260924 11:34:34.248124 13821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:34.249931 13720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:34:34.250105 13720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-2/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-2/wal
uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5"
format_stamp: "Formatted at 2026-09-24 11:34:34 on dist-test-slave-cq7c"
server_key: "81ad0f7f583314e612bf85f2d16c9a31"
server_key_iv: "87b892368ee0ff00b02cf4d61af490a3"
server_key_version: "encryptionkey@0"
I20260924 11:34:34.250288 13720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:34.266007 13720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:34.271162 13720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:34:34.271526 13720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:34.291229 13720 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.60:43695
I20260924 11:34:34.291302 13872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.60:43695 every 8 connection(s)
I20260924 11:34:34.292738 13720 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-2/data/info.pb
I20260924 11:34:34.295316 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 13720
I20260924 11:34:34.298336 13873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:34:34.307235 13873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:34:34 dist-test-slave-cq7c krb5kdc[13114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.212.60@KRBTEST.COM
Sep 24 11:34:34 dist-test-slave-cq7c krb5kdc[13114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249674, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.212.60@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM
I20260924 11:34:34.329506 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:34.310610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57076 (local address 127.6.212.62:42625)
0924 11:34:34.311795 (+ 1185us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:34.311801 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:34.311821 (+ 20us) server_negotiation.cc:407] Connection header received
0924 11:34:34.311875 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:34.311879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:34.311966 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:34.312074 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:34.313124 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:34.314386 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:34.316519 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:34.317221 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:34.317354 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:34.322267 (+ 4913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:34.322274 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:34.322281 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:34.322326 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:34.325924 (+ 3598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:34.326645 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:34.326651 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:34.326655 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:34.326819 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:34.327485 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:34.327490 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:34.327493 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:34.327811 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:34.327963 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:34.328882 (+ 919us) server_negotiation.cc:299] Negotiation successful
0924 11:34:34.329106 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1058,"thread_start_us":964,"threads_started":1}
I20260924 11:34:34.330250 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:34.297694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36670 (local address 127.6.212.60:43695)
0924 11:34:34.298784 (+ 1090us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:34.298799 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:34.298843 (+ 44us) server_negotiation.cc:407] Connection header received
0924 11:34:34.300630 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:34.300655 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:34.301087 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:34.301540 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:34.303265 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:34.306080 (+ 2815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:34.307421 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:34.308527 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:34.308718 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:34.321337 (+ 12619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:34.321347 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:34.321361 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:34.321397 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:34.325625 (+ 4228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:34.326586 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:34.326596 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:34.326599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:34.326793 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:34.327214 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:34.327221 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:34.327225 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:34.327954 (+ 729us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:34.328230 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:34.328581 (+ 351us) server_negotiation.cc:299] Negotiation successful
0924 11:34:34.328990 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":852,"thread_start_us":441,"threads_started":1}
I20260924 11:34:34.330904 13877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:34.310369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.0.0.1:57076)
0924 11:34:34.311254 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:34.311326 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:34.311565 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:34.312141 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:34.312146 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:34.312746 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:34.312981 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:34.312990 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:34.314550 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:34.314554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:34.316327 (+ 1773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:34.316341 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:34.317639 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:34.317648 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:34.317835 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:34.318568 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:34.318590 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:34.322044 (+ 3454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:34.326056 (+ 4012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:34.326069 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:34.326082 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:34.326475 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:34.326958 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:34.326961 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:34.326963 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:34.327110 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:34.327992 (+ 882us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:34.328001 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:34.328240 (+ 239us) client_negotiation.cc:769] Sending connection context
0924 11:34:34.328538 (+ 298us) client_negotiation.cc:240] Negotiation successful
0924 11:34:34.330608 (+ 2070us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"mutex_wait_us":1548,"thread_start_us":395,"threads_started":1}
I20260924 11:34:34.332479 13873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.212.61" port: 38525 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:34:34 dist-test-slave-cq7c krb5kdc[13114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249674, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.212.60@KRBTEST.COM for oryx/127.6.212.61@KRBTEST.COM
I20260924 11:34:34.346352 13877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:34.333327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:38525 (local address 127.0.0.1:39016)
0924 11:34:34.333561 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:34.333587 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:34.333714 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:34.334135 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:34.334138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:34.334453 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:34.334586 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:34.334593 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:34.336308 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:34.336312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:34.337214 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:34.337225 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:34.337854 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:34.337858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:34.338036 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:34.338665 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:34.338682 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:34.340653 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:34.343811 (+ 3158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:34.343816 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:34.343819 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:34.344470 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:34.344886 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:34.344889 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:34.344892 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:34.345099 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:34.345552 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:34.345555 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:34.345689 (+ 134us) client_negotiation.cc:769] Sending connection context
0924 11:34:34.345805 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 11:34:34.346091 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 11:34:34.346395 13879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:34.333264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39016 (local address 127.6.212.61:38525)
0924 11:34:34.333872 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:34.333876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:34.333891 (+ 15us) server_negotiation.cc:407] Connection header received
0924 11:34:34.333939 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:34.333942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:34.334008 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:34.334094 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:34.334710 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:34.336175 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:34.337343 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:34.337723 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:34.337817 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:34.340828 (+ 3011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:34.340832 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:34.340835 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:34.340865 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:34.343629 (+ 2764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:34.344619 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:34.344622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:34.344625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:34.344746 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:34.345200 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:34.345203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:34.345205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:34.345455 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:34.345560 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:34.345900 (+ 340us) server_negotiation.cc:299] Negotiation successful
0924 11:34:34.346118 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":405,"threads_started":1}
I20260924 11:34:34.347683 13873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.212.60" port: 43695 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:34:34 dist-test-slave-cq7c krb5kdc[13114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249674, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.212.60@KRBTEST.COM for oryx/127.6.212.60@KRBTEST.COM
I20260924 11:34:34.360598 13877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:34.348416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:43695 (local address 127.0.0.1:36680)
0924 11:34:34.348619 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:34.348641 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:34.348739 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:34.349108 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:34.349112 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:34.349503 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:34.349643 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:34.349651 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:34.350572 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:34.350576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:34.351531 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:34.351541 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:34.352216 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:34.352221 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:34.352386 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:34.353156 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:34.353175 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:34.355629 (+ 2454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:34.358351 (+ 2722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:34.358355 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:34.358358 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:34.358707 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:34.359051 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:34.359058 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:34.359065 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:34.359248 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:34.359763 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:34.359766 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:34.359934 (+ 168us) client_negotiation.cc:769] Sending connection context
0924 11:34:34.360054 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 11:34:34.360246 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260924 11:34:34.360672 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:34.348691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36680 (local address 127.6.212.60:43695)
0924 11:34:34.348863 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:34.348873 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:34.348887 (+ 14us) server_negotiation.cc:407] Connection header received
0924 11:34:34.348924 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:34.348927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:34.348976 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:34.349064 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:34.349761 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:34.350447 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:34.351653 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:34.352038 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:34.352138 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:34.355837 (+ 3699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:34.355842 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:34.355845 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:34.355874 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:34.358213 (+ 2339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:34.358805 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:34.358808 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:34.358810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:34.358927 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:34.359349 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:34.359352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:34.359354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:34.359663 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:34.359768 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:34.360140 (+ 372us) server_negotiation.cc:299] Negotiation successful
0924 11:34:34.360392 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":52}
I20260924 11:34:34.366176 13873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5: Bootstrap starting.
I20260924 11:34:34.369843 13873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5: Neither blocks nor log segments found. Creating new log.
I20260924 11:34:34.370910 13873 log.cc:824] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5: Log is configured to *not* fsync() on all Append() calls
I20260924 11:34:34.373728 13873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5: No bootstrap required, opened a new log
I20260924 11:34:34.381275 13873 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } peers { permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 38525 } } peers { permanent_uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 43695 } }
I20260924 11:34:34.381640 13873 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:34:34.381726 13873 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe9da2f8b0e4ed79c32eb5d554a06e5, State: Initialized, Role: FOLLOWER
I20260924 11:34:34.382251 13873 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } peers { permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 38525 } } peers { permanent_uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 43695 } }
I20260924 11:34:34.383059 13881 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } peers { permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 38525 } } peers { permanent_uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 43695 } } }
I20260924 11:34:34.383416 13881 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:34:34.384368 13873 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 11:34:34.394115 13892 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 11:34:34.394353 13892 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 11:34:34.397035 13720 master_runner.cc:428] Detected that this master bbe9da2f8b0e4ed79c32eb5d554a06e5 is joining an existing cluster
I20260924 11:34:34.397140 13720 master_runner.cc:432] Initiating AddMaster RPC to add 127.6.212.60:43695
I20260924 11:34:34.431769 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:34.401840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57084 (local address 127.6.212.62:42625)
0924 11:34:34.402055 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:34.402064 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:34.412607 (+ 10543us) server_negotiation.cc:407] Connection header received
0924 11:34:34.413415 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:34.413425 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:34.413533 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:34.413623 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:34.415591 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:34.417009 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:34.420649 (+ 3640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:34.421375 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:34.421475 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:34.424453 (+ 2978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:34.424460 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:34.424465 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:34.424512 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:34.428510 (+ 3998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:34.429291 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:34.429297 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:34.429301 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:34.429449 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:34.429921 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:34.429926 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:34.429929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:34.430255 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:34.430396 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:34.431107 (+ 711us) server_negotiation.cc:299] Negotiation successful
0924 11:34:34.431388 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 11:34:34.432834 13879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:34.401249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39028 (local address 127.6.212.61:38525)
0924 11:34:34.401693 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:34.401701 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:34.416057 (+ 14356us) server_negotiation.cc:407] Connection header received
0924 11:34:34.416301 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:34.416306 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:34.416396 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:34.416470 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:34.418552 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:34.419672 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:34.421914 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:34.422599 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:34.422700 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:34.425224 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:34.425231 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:34.425236 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:34.425277 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:34.429038 (+ 3761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:34.430230 (+ 1192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:34.430236 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:34.430239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:34.430388 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:34.430853 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:34.430858 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:34.430862 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:34.431212 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:34.431614 (+ 402us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:34.432247 (+ 633us) server_negotiation.cc:299] Negotiation successful
0924 11:34:34.432493 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291}
I20260924 11:34:34.433525 13898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:34.401138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.0.0.1:57084)
0924 11:34:34.412427 (+ 11289us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:34.412461 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:34.413196 (+ 735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:34.413905 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:34.413913 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:34.414859 (+ 946us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:34.415394 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:34.415407 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:34.417280 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:34.417289 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:34.419637 (+ 2348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:34.419661 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:34.421813 (+ 2152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:34.421821 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:34.422062 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:34.422815 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:34.422838 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:34.424228 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:34.428674 (+ 4446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:34.428682 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:34.428691 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:34.429154 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:34.429604 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:34.429610 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:34.429613 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:34.429793 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:34.430536 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:34.430546 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:34.430791 (+ 245us) client_negotiation.cc:769] Sending connection context
0924 11:34:34.432361 (+ 1570us) client_negotiation.cc:240] Negotiation successful
0924 11:34:34.432716 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11048,"thread_start_us":11501,"threads_started":1}
I20260924 11:34:34.434023 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:34.401639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:38525 (local address 127.0.0.1:39028)
0924 11:34:34.415927 (+ 14288us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:34.415958 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:34.416153 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:34.416728 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:34.416732 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:34.417518 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:34.418337 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:34.418351 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:34.419869 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:34.419875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:34.421719 (+ 1844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:34.421734 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:34.422978 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:34.422985 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:34.423168 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:34.423853 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:34.423886 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:34.425016 (+ 1130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:34.429224 (+ 4208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:34.429230 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:34.429235 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:34.430096 (+ 861us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:34.430524 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:34.430530 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:34.430535 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:34.430705 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:34.431727 (+ 1022us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:34.431732 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:34.431950 (+ 218us) client_negotiation.cc:769] Sending connection context
0924 11:34:34.433115 (+ 1165us) client_negotiation.cc:240] Negotiation successful
0924 11:34:34.433558 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14146,"thread_start_us":14541,"threads_started":1}
I20260924 11:34:34.437542 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 13220
I20260924 11:34:34.438596 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:34.401767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:43695 (local address 127.0.0.1:36688)
0924 11:34:34.420279 (+ 18512us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:34.420307 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:34.420454 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:34.420875 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:34.420880 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:34.421576 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:34.421951 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:34.421962 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:34.423462 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:34.423467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:34.425164 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:34.425179 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:34.426886 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:34.426895 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:34.427082 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:34.427653 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:34.427690 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:34.428795 (+ 1105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:34.433296 (+ 4501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:34.433300 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:34.433305 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:34.433887 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:34.434303 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:34.434307 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:34.434310 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:34.434463 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:34.435623 (+ 1160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:34.435628 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:34.435779 (+ 151us) client_negotiation.cc:769] Sending connection context
0924 11:34:34.437311 (+ 1532us) client_negotiation.cc:240] Negotiation successful
0924 11:34:34.437916 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18367,"thread_start_us":18202,"threads_started":1}
I20260924 11:34:34.439189 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:34.401209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36688 (local address 127.6.212.60:43695)
0924 11:34:34.401411 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:34.401420 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:34.420401 (+ 18981us) server_negotiation.cc:407] Connection header received
0924 11:34:34.420591 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:34.420598 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:34.420667 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:34.420751 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:34.422093 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:34.423230 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:34.425322 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:34.426448 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:34.426568 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:34.428954 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:34.428960 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:34.428964 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:34.429000 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:34.433146 (+ 4146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:34.434019 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:34.434025 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:34.434028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:34.434183 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:34.434587 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:34.434592 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:34.434595 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:34.434973 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:34.435111 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:34.437024 (+ 1913us) server_negotiation.cc:299] Negotiation successful
0924 11:34:34.438864 (+ 1840us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Sep 24 11:34:34 dist-test-slave-cq7c krb5kdc[13114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.212.1@KRBTEST.COM
Sep 24 11:34:34 dist-test-slave-cq7c krb5kdc[13114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249671, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.212.62@KRBTEST.COM for oryx/127.6.212.60@KRBTEST.COM
I20260924 11:34:34.460135 13715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:34.442750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:43695 (local address 127.0.0.1:36692)
0924 11:34:34.443055 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:34.443073 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:34.443188 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:34.443665 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:34.443670 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:34.444117 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:34.444283 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:34.444292 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:34.445748 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:34.445754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:34.447118 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:34.447134 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:34.448722 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:34.448729 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:34.448934 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:34.449560 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:34.449583 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:34.452574 (+ 2991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:34.457002 (+ 4428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:34.457007 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:34.457012 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:34.457590 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:34.458067 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:34.458072 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:34.458075 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:34.458254 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:34.459028 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:34.459033 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:34.459197 (+ 164us) client_negotiation.cc:769] Sending connection context
0924 11:34:34.459386 (+ 189us) client_negotiation.cc:240] Negotiation successful
0924 11:34:34.459716 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184}
I20260924 11:34:34.461102 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:34.442802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36692 (local address 127.6.212.60:43695)
0924 11:34:34.442950 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:34.442955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:34.443156 (+ 201us) server_negotiation.cc:407] Connection header received
0924 11:34:34.443321 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:34.443324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:34.443479 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:34.443598 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:34.444435 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:34.445586 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:34.447276 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:34.448545 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:34.448688 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:34.452801 (+ 4113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:34.452808 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:34.452813 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:34.452857 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:34.456791 (+ 3934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:34.457746 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:34.457752 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:34.457756 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:34.457903 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:34.458376 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:34.458381 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:34.458385 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:34.458793 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:34.458928 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:34.460460 (+ 1532us) server_negotiation.cc:299] Negotiation successful
0924 11:34:34.460710 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 11:34:34.461944 13901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:34.438943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48898 (local address 127.6.212.1:42317)
0924 11:34:34.440502 (+ 1559us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:34.440512 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:34.440542 (+ 30us) server_negotiation.cc:407] Connection header received
0924 11:34:34.440609 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:34.440618 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:34.440907 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:34.441077 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:34.442659 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:34.444417 (+ 1758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:34.446199 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:34.446814 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:34.446970 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:34.454050 (+ 7080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:34.454068 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:34.454085 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:34.454120 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:34.458627 (+ 4507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:34.459415 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:34.459432 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:34.459435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:34.459570 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:34.459993 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:34.459998 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:34.460000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:34.460525 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:34.460845 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:34.461275 (+ 430us) server_negotiation.cc:299] Negotiation successful
0924 11:34:34.461552 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1367,"thread_start_us":1756,"threads_started":1}
I20260924 11:34:34.462766 13159 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.6.212.60:43695: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5"
member_type: NON_VOTER
last_known_addr {
host: "127.6.212.60"
port: 43695
}
attrs {
promote: true
}
}
dest_uuid: "f32c56b9b2b94674b1fde17e4e460006"
cas_config_opid_index: 5
I20260924 11:34:34.463793 13159 consensus_queue.cc:237] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } peers { permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: NON_VOTER last_known_addr { host: "127.6.212.61" port: 38525 } attrs { promote: true } } peers { permanent_uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5" member_type: NON_VOTER last_known_addr { host: "127.6.212.60" port: 43695 } attrs { promote: true } }
I20260924 11:34:34.464203 13343 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:34:34.465955 13717 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER bbe9da2f8b0e4ed79c32eb5d554a06e5 (127.6.212.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } peers { permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: NON_VOTER last_known_addr { host: "127.6.212.61" port: 38525 } attrs { promote: true } } peers { permanent_uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5" member_type: NON_VOTER last_known_addr { host: "127.6.212.60" port: 43695 } attrs { promote: true } } }
I20260924 11:34:34.471240 13717 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.6.212.60:43695
I20260924 11:34:34.474298 13848 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:34:34.481841 13720 master.cc:537] Master@127.6.212.60:43695 shutting down...
I20260924 11:34:34.483347 13848 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5 [term 1 FOLLOWER]: Refusing update from remote peer f32c56b9b2b94674b1fde17e4e460006: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 11:34:34.486002 13903 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5" member_type: NON_VOTER last_known_addr { host: "127.6.212.60" port: 43695 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:34:34.488790 13904 sys_catalog.cc:455] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f32c56b9b2b94674b1fde17e4e460006" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } peers { permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: NON_VOTER last_known_addr { host: "127.6.212.61" port: 38525 } attrs { promote: true } } peers { permanent_uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5" member_type: NON_VOTER last_known_addr { host: "127.6.212.60" port: 43695 } attrs { promote: true } } }
I20260924 11:34:34.489355 13904 sys_catalog.cc:458] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 [sys.catalog]: This master's current role is: LEADER
W20260924 11:34:34.495360 13145 consensus_peers.cc:487] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 -> Peer bbe9da2f8b0e4ed79c32eb5d554a06e5 (127.6.212.60:43695): Couldn't send request to peer bbe9da2f8b0e4ed79c32eb5d554a06e5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 11:34:34.499711 13715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:34.471819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:38525 (local address 127.0.0.1:39034)
0924 11:34:34.485313 (+ 13494us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:34.485333 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:34.485489 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:34.486313 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:34.486318 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:34.486350 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:34:34.486600 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:34.486613 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:34.488579 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:34.488586 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:34.496987 (+ 8401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:34.497027 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:34.498954 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:34.498959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:34.499112 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:34.499151 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 11:34:34.499268 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 11:34:34.499449 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13310}
I20260924 11:34:34.499730 13879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:34.472463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39034 (local address 127.6.212.61:38525)
0924 11:34:34.472729 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:34.472738 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:34.485424 (+ 12686us) server_negotiation.cc:407] Connection header received
0924 11:34:34.485627 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:34.485635 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:34.485746 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:34.485860 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:34:34.487231 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:34.488346 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:34.497264 (+ 8918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:34.498564 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:34.498662 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:34.498832 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:34.499297 (+ 465us) server_negotiation.cc:299] Negotiation successful
0924 11:34:34.499449 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 11:34:34.500849 13767 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [term 1 LEARNER]: Refusing update from remote peer f32c56b9b2b94674b1fde17e4e460006: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 11:34:34.501734 13903 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: NON_VOTER last_known_addr { host: "127.6.212.61" port: 38525 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 11:34:34.507424 13767 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER bbe9da2f8b0e4ed79c32eb5d554a06e5 (127.6.212.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } peers { permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: NON_VOTER last_known_addr { host: "127.6.212.61" port: 38525 } attrs { promote: true } } peers { permanent_uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5" member_type: NON_VOTER last_known_addr { host: "127.6.212.60" port: 43695 } attrs { promote: true } } }
I20260924 11:34:34.511808 13795 sys_catalog.cc:455] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f32c56b9b2b94674b1fde17e4e460006. Latest consensus state: current_term: 1 leader_uuid: "f32c56b9b2b94674b1fde17e4e460006" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } peers { permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: NON_VOTER last_known_addr { host: "127.6.212.61" port: 38525 } attrs { promote: true } } peers { permanent_uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5" member_type: NON_VOTER last_known_addr { host: "127.6.212.60" port: 43695 } attrs { promote: true } } }
I20260924 11:34:34.512077 13795 sys_catalog.cc:458] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [sys.catalog]: This master's current role is: LEARNER
I20260924 11:34:34.512603 13904 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006: attempting to promote NON_VOTER d211d01c60f2417bb346495d00f5593f to VOTER
I20260924 11:34:34.513340 13904 consensus_queue.cc:237] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } peers { permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 38525 } attrs { promote: false } } peers { permanent_uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5" member_type: NON_VOTER last_known_addr { host: "127.6.212.60" port: 43695 } attrs { promote: true } }
I20260924 11:34:34.515484 13767 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [term 1 LEARNER]: Refusing update from remote peer f32c56b9b2b94674b1fde17e4e460006: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 11:34:34.516053 13717 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 38525 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260924 11:34:34.516271 13145 consensus_peers.cc:487] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 -> Peer bbe9da2f8b0e4ed79c32eb5d554a06e5 (127.6.212.60:43695): Couldn't send request to peer bbe9da2f8b0e4ed79c32eb5d554a06e5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 11:34:34.516494 13907 sys_catalog.cc:455] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f32c56b9b2b94674b1fde17e4e460006" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } peers { permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: NON_VOTER last_known_addr { host: "127.6.212.61" port: 38525 } attrs { promote: true } } peers { permanent_uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5" member_type: NON_VOTER last_known_addr { host: "127.6.212.60" port: 43695 } attrs { promote: true } } }
I20260924 11:34:34.516920 13907 sys_catalog.cc:458] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [sys.catalog]: This master's current role is: LEARNER
I20260924 11:34:34.518536 13903 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d211d01c60f2417bb346495d00f5593f (127.6.212.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } peers { permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 38525 } attrs { promote: false } } peers { permanent_uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5" member_type: NON_VOTER last_known_addr { host: "127.6.212.60" port: 43695 } attrs { promote: true } } }
I20260924 11:34:34.519207 13766 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d211d01c60f2417bb346495d00f5593f (127.6.212.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } peers { permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 38525 } attrs { promote: false } } peers { permanent_uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5" member_type: NON_VOTER last_known_addr { host: "127.6.212.60" port: 43695 } attrs { promote: true } } }
I20260924 11:34:34.522662 13903 sys_catalog.cc:455] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f32c56b9b2b94674b1fde17e4e460006" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } peers { permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 38525 } attrs { promote: false } } peers { permanent_uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5" member_type: NON_VOTER last_known_addr { host: "127.6.212.60" port: 43695 } attrs { promote: true } } }
I20260924 11:34:34.522603 13717 sys_catalog.cc:455] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f32c56b9b2b94674b1fde17e4e460006" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } peers { permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 38525 } attrs { promote: false } } peers { permanent_uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5" member_type: NON_VOTER last_known_addr { host: "127.6.212.60" port: 43695 } attrs { promote: true } } }
I20260924 11:34:34.522954 13903 sys_catalog.cc:458] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 [sys.catalog]: This master's current role is: LEADER
I20260924 11:34:34.522951 13717 sys_catalog.cc:458] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 [sys.catalog]: This master's current role is: LEADER
W20260924 11:34:34.523716 13827 tablet.cc:2369] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:34:34.524654 13907 sys_catalog.cc:455] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f32c56b9b2b94674b1fde17e4e460006" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } peers { permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 38525 } attrs { promote: false } } peers { permanent_uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5" member_type: NON_VOTER last_known_addr { host: "127.6.212.60" port: 43695 } attrs { promote: true } } }
I20260924 11:34:34.524945 13907 sys_catalog.cc:458] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:34:34.526674 13720 raft_consensus.cc:2229] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:34:34.527079 13720 raft_consensus.cc:2258] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:34:34.527326 13720 tablet_replica.cc:330] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5: stopping tablet replica
I20260924 11:34:34.543231 13720 master.cc:552] Master@127.6.212.60:43695 shutdown complete.
I20260924 11:34:34.543350 13720 master_runner.cc:305] Clearing existing system tablet
I20260924 11:34:34.543841 13720 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:34.543975 13720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:34.546303 13720 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 11:34:34.547822 13909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:34.548369 13720 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:34.548625 13720 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 11:34:34.548729 13720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-2/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-2/wal
uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5"
format_stamp: "Formatted at 2026-09-24 11:34:34 on dist-test-slave-cq7c"
server_key: "81ad0f7f583314e612bf85f2d16c9a31"
server_key_iv: "87b892368ee0ff00b02cf4d61af490a3"
server_key_version: "encryptionkey@0"
I20260924 11:34:34.552461 13720 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:34:34.557392 13720 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 11:34:34.557574 13720 log.cc:1196] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 11:34:34.557996 13720 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5: Deleting consensus metadata
I20260924 11:34:34.558272 13720 master_runner.cc:315] Copying system tablet from 127.6.212.62:42625
I20260924 11:34:34.560981 13720 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5: tablet copy: Beginning tablet copy session from remote peer at address 127.6.212.62:42625
I20260924 11:34:34.576488 13878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:34.561879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57092 (local address 127.6.212.62:42625)
0924 11:34:34.562156 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:34.562166 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:34.564018 (+ 1852us) server_negotiation.cc:407] Connection header received
0924 11:34:34.564251 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:34.564257 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:34.564380 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:34.564481 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:34.565336 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:34.566104 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:34.567810 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:34.568418 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:34.568572 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:34.570547 (+ 1975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:34.570558 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:34.570561 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:34.570594 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:34.573764 (+ 3170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:34.574440 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:34.574446 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:34.574450 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:34.574599 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:34.574983 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:34.574990 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:34.574993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:34.575271 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:34.575453 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:34.575884 (+ 431us) server_negotiation.cc:299] Negotiation successful
0924 11:34:34.576107 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260924 11:34:34.576553 13915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:34.561855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.0.0.1:57092)
0924 11:34:34.563891 (+ 2036us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:34.563909 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:34.564099 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:34.564507 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:34.564511 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:34.564977 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:34.565183 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:34.565193 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:34.566261 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:34.566266 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:34.567669 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:34.567683 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:34.568589 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:34.568595 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:34.568763 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:34.569528 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:34.569548 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:34.570329 (+ 781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:34.573916 (+ 3587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:34.573921 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:34.573925 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:34.574306 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:34.574712 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:34.574718 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:34.574722 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:34.574875 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:34.575438 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:34.575443 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:34.575631 (+ 188us) client_negotiation.cc:769] Sending connection context
0924 11:34:34.575774 (+ 143us) client_negotiation.cc:240] Negotiation successful
0924 11:34:34.576129 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1910,"thread_start_us":1714,"threads_started":1}
I20260924 11:34:34.577441 13179 tablet_copy_service.cc:140] P f32c56b9b2b94674b1fde17e4e460006: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer bbe9da2f8b0e4ed79c32eb5d554a06e5 ({username='oryx', principal='oryx/127.6.212.60@KRBTEST.COM'} at 127.0.0.1:57092)
I20260924 11:34:34.577630 13179 tablet_copy_service.cc:161] P f32c56b9b2b94674b1fde17e4e460006: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer bbe9da2f8b0e4ed79c32eb5d554a06e5 at {username='oryx', principal='oryx/127.6.212.60@KRBTEST.COM'} at 127.0.0.1:57092: session id = bbe9da2f8b0e4ed79c32eb5d554a06e5-00000000000000000000000000000000
I20260924 11:34:34.580040 13179 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 11:34:34.581719 13720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 11:34:34.587353 13720 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5: tablet copy: Starting download of 0 data blocks...
I20260924 11:34:34.587720 13720 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5: tablet copy: Starting download of 1 WAL segments...
I20260924 11:34:34.590245 13720 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 11:34:34.593909 13179 tablet_copy_service.cc:342] P f32c56b9b2b94674b1fde17e4e460006: Request end of tablet copy session bbe9da2f8b0e4ed79c32eb5d554a06e5-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.6.212.60@KRBTEST.COM'} at 127.0.0.1:57092
I20260924 11:34:34.594069 13179 tablet_copy_service.cc:433] P f32c56b9b2b94674b1fde17e4e460006: ending tablet copy session bbe9da2f8b0e4ed79c32eb5d554a06e5-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer bbe9da2f8b0e4ed79c32eb5d554a06e5
I20260924 11:34:34.605119 13720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:34.609077 13917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:34.611814 13916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:34.618077 13720 server_base.cc:1034] running on GCE node
W20260924 11:34:34.618729 13919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:34.619257 13720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:34.619918 13720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:34.621106 13720 hybrid_clock.cc:648] HybridClock initialized: now 1790249674621037 us; error 63 us; skew 500 ppm
I20260924 11:34:34.621357 13720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:34:34 dist-test-slave-cq7c krb5kdc[13114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249674, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.212.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:34.623268 13720 init.cc:377] Logged in from keytab as oryx/127.6.212.60@KRBTEST.COM (short username oryx)
I20260924 11:34:34.628146 13720 webserver.cc:460] Webserver started at http://127.6.212.60:38857/ using document root <none> and password file <none>
I20260924 11:34:34.628855 13720 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:34.628973 13720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:34.631338 13720 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260924 11:34:34.633419 13926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:34.633822 13720 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20260924 11:34:34.633962 13720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-2/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-2/wal
uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5"
format_stamp: "Formatted at 2026-09-24 11:34:34 on dist-test-slave-cq7c"
server_key: "81ad0f7f583314e612bf85f2d16c9a31"
server_key_iv: "87b892368ee0ff00b02cf4d61af490a3"
server_key_version: "encryptionkey@0"
I20260924 11:34:34.634158 13720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:34.640733 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 13220
I20260924 11:34:34.643196 13720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:34.646802 13720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:34.666326 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.1:42317
--local_ip_for_outbound_sockets=127.6.212.1
--tserver_master_addrs=127.6.212.62:42625
--webserver_port=45811
--webserver_interface=127.6.212.1
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.1.keytab
--principal=oryx/127.6.212.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:33661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.212.62:42625,127.6.212.61:38525
--tserver_master_addrs=127.6.212.62:42625,127.6.212.61:38525,127.6.212.60:43695 with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:34:34.678711 13720 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.60:43695
I20260924 11:34:34.678733 13978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.60:43695 every 8 connection(s)
I20260924 11:34:34.679714 13720 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/master-2/data/info.pb
I20260924 11:34:34.686743 13979 sys_catalog.cc:263] Verifying existing consensus state
I20260924 11:34:34.692242 13979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5: Bootstrap starting.
I20260924 11:34:34.712687 13979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 11:34:34.713197 13979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5: Bootstrap complete.
I20260924 11:34:34.714087 13979 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } peers { permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 38525 } attrs { promote: false } } peers { permanent_uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5" member_type: NON_VOTER last_known_addr { host: "127.6.212.60" port: 43695 } attrs { promote: true } }
I20260924 11:34:34.714336 13979 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bbe9da2f8b0e4ed79c32eb5d554a06e5, State: Initialized, Role: LEARNER
I20260924 11:34:34.714583 13979 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } peers { permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 38525 } attrs { promote: false } } peers { permanent_uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5" member_type: NON_VOTER last_known_addr { host: "127.6.212.60" port: 43695 } attrs { promote: true } }
I20260924 11:34:34.716048 13982 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } peers { permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 38525 } attrs { promote: false } } peers { permanent_uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5" member_type: NON_VOTER last_known_addr { host: "127.6.212.60" port: 43695 } attrs { promote: true } } }
I20260924 11:34:34.716427 13979 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:34:34.716589 13982 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5 [sys.catalog]: This master's current role is: LEARNER
I20260924 11:34:34.725339 13993 catalog_manager.cc:1259] Loaded cluster ID: 667ded6d01e14754a126c1667f74f666
I20260924 11:34:34.725474 13993 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5: loading cluster ID for follower catalog manager: success
I20260924 11:34:34.728271 13993 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5: acquiring CA information for follower catalog manager: success
I20260924 11:34:34.730515 13993 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 11:34:35.080084 13965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:35.080390 13965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:35.080587 13965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:35.088409 13965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:35.088546 13965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:35.088711 13965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.1
I20260924 11:34:35.099078 13965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:33661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.1.keytab
--principal=oryx/127.6.212.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.1:42317
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.212.1
--webserver_port=45811
--tserver_master_addrs=127.6.212.62:42625,127.6.212.61:38525,127.6.212.60:43695
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.1
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:34:35.100839 13965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:35.102967 13965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:35.121841 14000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:35.122131 14002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:35.123059 13999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:35.123513 13965 server_base.cc:1034] running on GCE node
I20260924 11:34:35.124262 13965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:35.125440 13965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:35.126669 13965 hybrid_clock.cc:648] HybridClock initialized: now 1790249675126610 us; error 51 us; skew 500 ppm
I20260924 11:34:35.126997 13965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:34:35.129290 14003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:35.112298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:43695 (local address 127.0.0.1:36694)
0924 11:34:35.122463 (+ 10165us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:35.122503 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:35.122677 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:35.123284 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:35.123290 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:35.123320 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:34:35.123567 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:35.123577 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:35.125371 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:35.125377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:35.127035 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:35.127048 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:35.128656 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:35.128661 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:35.128790 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:35.128816 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 11:34:35.128899 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 11:34:35.129077 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9633,"thread_start_us":10136,"threads_started":1}
I20260924 11:34:35.129362 14004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:35.112585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36694 (local address 127.6.212.60:43695)
0924 11:34:35.121798 (+ 9213us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:35.121807 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:35.122609 (+ 802us) server_negotiation.cc:407] Connection header received
0924 11:34:35.122962 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:35.122968 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:35.123102 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:35.123235 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:34:35.123753 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:35.125201 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:35.127196 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:35.128460 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:35.128612 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:35.128836 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:35.128923 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 11:34:35.129056 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9029,"thread_start_us":9768,"threads_started":1}
Sep 24 11:34:35 dist-test-slave-cq7c krb5kdc[13114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249675, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.212.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:35.130330 13953 raft_consensus.cc:1203] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260924 11:34:35.131069 13965 init.cc:377] Logged in from keytab as oryx/127.6.212.1@KRBTEST.COM (short username oryx)
I20260924 11:34:35.130963 13982 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f32c56b9b2b94674b1fde17e4e460006. Latest consensus state: current_term: 1 leader_uuid: "f32c56b9b2b94674b1fde17e4e460006" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } peers { permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 38525 } attrs { promote: false } } peers { permanent_uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5" member_type: NON_VOTER last_known_addr { host: "127.6.212.60" port: 43695 } attrs { promote: true } } }
I20260924 11:34:35.131253 13982 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5 [sys.catalog]: This master's current role is: LEARNER
I20260924 11:34:35.132793 13965 webserver.cc:460] Webserver started at http://127.6.212.1:45811/ using document root <none> and password file <none>
I20260924 11:34:35.133697 13965 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:35.133822 13965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:35.139716 13965 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 11:34:35.142884 14011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:35.144610 13965 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 11:34:35.144801 13965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-0/wal
uuid: "92c42ff207f3406aa06f3984a03fc08b"
format_stamp: "Formatted at 2026-09-24 11:34:31 on dist-test-slave-cq7c"
server_key: "0a986ebfcc227100070d3b60250e194f"
server_key_iv: "a3cd4569096dcfe3625dfad6fc8e7318"
server_key_version: "encryptionkey@0"
I20260924 11:34:35.145960 13965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:35.175258 13965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:35.179652 13965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:34:35.179937 13965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:35.180969 13965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:34:35.182439 13965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:34:35.182572 13965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:35.182700 13965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:34:35.182884 13965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:35.222214 13965 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.1:42317
I20260924 11:34:35.222302 14123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.1:42317 every 8 connection(s)
I20260924 11:34:35.223955 13965 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
I20260924 11:34:35.226186 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 13965
I20260924 11:34:35.226532 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 13354
Sep 24 11:34:35 dist-test-slave-cq7c krb5kdc[13114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.212.2@KRBTEST.COM
Sep 24 11:34:35 dist-test-slave-cq7c krb5kdc[13114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249675, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.212.1@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM
Sep 24 11:34:35 dist-test-slave-cq7c krb5kdc[13114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249675, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.212.1@KRBTEST.COM for oryx/127.6.212.60@KRBTEST.COM
Sep 24 11:34:35 dist-test-slave-cq7c krb5kdc[13114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249675, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.212.1@KRBTEST.COM for oryx/127.6.212.61@KRBTEST.COM
I20260924 11:34:35.273160 14131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:35.227814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55620 (local address 127.6.212.2:34887)
0924 11:34:35.249418 (+ 21604us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:35.249429 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:35.249506 (+ 77us) server_negotiation.cc:407] Connection header received
0924 11:34:35.249588 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:35.249597 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:35.249875 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:35.250060 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:35.252401 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:35.253971 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:35.255293 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:35.256357 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:35.256519 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:35.264292 (+ 7773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:35.264315 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:35.264348 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:35.264405 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:35.268323 (+ 3918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:35.269592 (+ 1269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:35.269604 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:35.269607 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:35.269777 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:35.270261 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:35.270267 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:35.270272 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:35.270788 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:35.271105 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:35.272405 (+ 1300us) server_negotiation.cc:299] Negotiation successful
0924 11:34:35.272699 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21375,"thread_start_us":21234,"threads_started":1}
I20260924 11:34:35.274875 13477 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:34:35.276657 14004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:35.239585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:35141 (local address 127.6.212.60:43695)
0924 11:34:35.239824 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:35.239833 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:35.250954 (+ 11121us) server_negotiation.cc:407] Connection header received
0924 11:34:35.251111 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:35.251120 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:35.251224 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:35.251341 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:35.255865 (+ 4524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:35.256976 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:35.260883 (+ 3907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:35.261574 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:35.261697 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:35.268345 (+ 6648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:35.268352 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:35.268357 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:35.268408 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:35.272243 (+ 3835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:35.273622 (+ 1379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:35.273627 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:35.273631 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:35.273781 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:35.274296 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:35.274301 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:35.274304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:35.274670 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:35.274773 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:35.275962 (+ 1189us) server_negotiation.cc:299] Negotiation successful
0924 11:34:35.276295 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 11:34:35.276746 14129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:35.227260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:59353 (local address 127.6.212.62:42625)
0924 11:34:35.253364 (+ 26104us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:35.253371 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:35.253397 (+ 26us) server_negotiation.cc:407] Connection header received
0924 11:34:35.253454 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:35.253459 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:35.253534 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:35.253643 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:35.256366 (+ 2723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:35.257474 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:35.260883 (+ 3409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:35.261564 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:35.261712 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:35.268156 (+ 6444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:35.268163 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:35.268169 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:35.268220 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:35.272380 (+ 4160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:35.273517 (+ 1137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:35.273526 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:35.273530 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:35.273682 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:35.274334 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:35.274340 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:35.274343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:35.274773 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:35.274992 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:35.276141 (+ 1149us) server_negotiation.cc:299] Negotiation successful
0924 11:34:35.276368 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25975,"thread_start_us":25750,"threads_started":1}
I20260924 11:34:35.279053 14133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:35.238871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:43695 (local address 127.6.212.1:35141)
0924 11:34:35.250824 (+ 11953us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:35.250846 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:35.250989 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:35.251997 (+ 1008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:35.252013 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:35.253098 (+ 1085us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:35.255586 (+ 2488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:35.255623 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:35.257186 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:35.257191 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:35.260604 (+ 3413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:35.260620 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:35.261984 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:35.261989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:35.262377 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:35.263638 (+ 1261us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:35.263667 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:35.268089 (+ 4422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:35.272421 (+ 4332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:35.272431 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:35.272450 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:35.273479 (+ 1029us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:35.273926 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:35.273930 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:35.273933 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:35.274159 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:35.274844 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:35.274849 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:35.275455 (+ 606us) client_negotiation.cc:769] Sending connection context
0924 11:34:35.276889 (+ 1434us) client_negotiation.cc:240] Negotiation successful
0924 11:34:35.277554 (+ 665us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11803,"mutex_wait_us":258,"spinlock_wait_cycles":32128,"thread_start_us":8739,"threads_started":1}
I20260924 11:34:35.280046 14134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:35.228413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.6.212.1:59353)
0924 11:34:35.247966 (+ 19553us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:35.247995 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:35.248167 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:35.253695 (+ 5528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:35.253700 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:35.255882 (+ 2182us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:35.256165 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:35.256176 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:35.257641 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:35.257649 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:35.260604 (+ 2955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:35.260620 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:35.261969 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:35.261978 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:35.262482 (+ 504us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:35.263296 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:35.263338 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:35.267739 (+ 4401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:35.272579 (+ 4840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:35.272587 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:35.272593 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:35.273308 (+ 715us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:35.273854 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:35.273861 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:35.273868 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:35.274186 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:35.274944 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:35.274948 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:35.275455 (+ 507us) client_negotiation.cc:769] Sending connection context
0924 11:34:35.276964 (+ 1509us) client_negotiation.cc:240] Negotiation successful
0924 11:34:35.279578 (+ 2614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19362,"mutex_wait_us":2615,"spinlock_wait_cycles":26624,"thread_start_us":22059,"threads_started":1}
I20260924 11:34:35.281383 14124 heartbeater.cc:347] Connected to a master server at 127.6.212.60:43695
I20260924 11:34:35.281483 14125 heartbeater.cc:347] Connected to a master server at 127.6.212.62:42625
I20260924 11:34:35.281778 14125 heartbeater.cc:464] Registering TS with master...
I20260924 11:34:35.281790 14124 heartbeater.cc:464] Registering TS with master...
I20260924 11:34:35.282907 14132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:35.227778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:38525 (local address 127.6.212.1:44899)
0924 11:34:35.240006 (+ 12228us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:35.240087 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:35.243787 (+ 3700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:35.252073 (+ 8286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:35.252080 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:35.253105 (+ 1025us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:35.255650 (+ 2545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:35.255674 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:35.257314 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:35.257327 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:35.260592 (+ 3265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:35.260617 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:35.262383 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:35.262391 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:35.262654 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:35.263460 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:35.263490 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:35.268507 (+ 5017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:35.273031 (+ 4524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:35.273037 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:35.273042 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:35.273733 (+ 691us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:35.274214 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:35.274221 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:35.274228 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:35.280466 (+ 6238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:35.281546 (+ 1080us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:35.281551 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:35.281778 (+ 227us) client_negotiation.cc:769] Sending connection context
0924 11:34:35.281940 (+ 162us) client_negotiation.cc:240] Negotiation successful
0924 11:34:35.282286 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11738,"mutex_wait_us":814,"thread_start_us":16443,"threads_started":1}
I20260924 11:34:35.283279 14125 heartbeater.cc:511] Master 127.6.212.62:42625 requested a full tablet report, sending...
I20260924 11:34:35.283529 14124 heartbeater.cc:511] Master 127.6.212.60:43695 requested a full tablet report, sending...
I20260924 11:34:35.284103 14130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:35.227711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:44899 (local address 127.6.212.61:38525)
0924 11:34:35.251534 (+ 23823us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:35.251542 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:35.251564 (+ 22us) server_negotiation.cc:407] Connection header received
0924 11:34:35.251635 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:35.251642 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:35.251721 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:35.251834 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:35.255988 (+ 4154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:35.257069 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:35.260938 (+ 3869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:35.261890 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:35.262026 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:35.268783 (+ 6757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:35.268791 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:35.268797 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:35.268844 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:35.272829 (+ 3985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:35.273903 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:35.273912 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:35.273917 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:35.274069 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:35.280700 (+ 6631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:35.280708 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:35.280712 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:35.281264 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:35.281473 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:35.283318 (+ 1845us) server_negotiation.cc:299] Negotiation successful
0924 11:34:35.283656 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23676,"thread_start_us":23543,"threads_started":1}
I20260924 11:34:35.285157 14126 heartbeater.cc:347] Connected to a master server at 127.6.212.61:38525
I20260924 11:34:35.285327 14126 heartbeater.cc:464] Registering TS with master...
I20260924 11:34:35.285776 14126 heartbeater.cc:511] Master 127.6.212.61:38525 requested a full tablet report, sending...
I20260924 11:34:35.285743 13158 ts_manager.cc:194] Re-registered known tserver with Master: 92c42ff207f3406aa06f3984a03fc08b (127.6.212.1:42317)
I20260924 11:34:35.286047 13943 ts_manager.cc:194] Registered new tserver with Master: 92c42ff207f3406aa06f3984a03fc08b (127.6.212.1:42317)
I20260924 11:34:35.286968 13158 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.212.1@KRBTEST.COM'} at 127.6.212.1:59353
I20260924 11:34:35.287647 13757 ts_manager.cc:194] Registered new tserver with Master: 92c42ff207f3406aa06f3984a03fc08b (127.6.212.1:42317)
I20260924 11:34:35.378038 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 13354
I20260924 11:34:35.394508 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.2:34887
--local_ip_for_outbound_sockets=127.6.212.2
--tserver_master_addrs=127.6.212.62:42625
--webserver_port=34395
--webserver_interface=127.6.212.2
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.2.keytab
--principal=oryx/127.6.212.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:33661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.212.62:42625,127.6.212.61:38525
--tserver_master_addrs=127.6.212.62:42625,127.6.212.61:38525,127.6.212.60:43695 with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:34:35.663944 13997 sys_catalog.cc:455] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f32c56b9b2b94674b1fde17e4e460006" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } peers { permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 38525 } attrs { promote: false } } peers { permanent_uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5" member_type: NON_VOTER last_known_addr { host: "127.6.212.60" port: 43695 } attrs { promote: true } } }
I20260924 11:34:35.664409 13997 sys_catalog.cc:458] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 [sys.catalog]: This master's current role is: LEADER
I20260924 11:34:35.665671 13997 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006: attempting to promote NON_VOTER bbe9da2f8b0e4ed79c32eb5d554a06e5 to VOTER
I20260924 11:34:35.666231 13997 consensus_queue.cc:237] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } peers { permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 38525 } attrs { promote: false } } peers { permanent_uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 43695 } attrs { promote: false } }
I20260924 11:34:35.668572 13766 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [term 1 FOLLOWER]: Refusing update from remote peer f32c56b9b2b94674b1fde17e4e460006: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260924 11:34:35.668622 13953 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5 [term 1 LEARNER]: Refusing update from remote peer f32c56b9b2b94674b1fde17e4e460006: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260924 11:34:35.669253 14138 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 38525 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 11:34:35.669739 14137 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 43695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 11:34:35.673031 14137 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bbe9da2f8b0e4ed79c32eb5d554a06e5 (127.6.212.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } peers { permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 38525 } attrs { promote: false } } peers { permanent_uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 43695 } attrs { promote: false } } }
I20260924 11:34:35.673875 13766 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bbe9da2f8b0e4ed79c32eb5d554a06e5 (127.6.212.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } peers { permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 38525 } attrs { promote: false } } peers { permanent_uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 43695 } attrs { promote: false } } }
I20260924 11:34:35.676959 14137 sys_catalog.cc:455] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f32c56b9b2b94674b1fde17e4e460006" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } peers { permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 38525 } attrs { promote: false } } peers { permanent_uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 43695 } attrs { promote: false } } }
I20260924 11:34:35.677232 14137 sys_catalog.cc:458] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 [sys.catalog]: This master's current role is: LEADER
I20260924 11:34:35.677294 13997 sys_catalog.cc:455] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f32c56b9b2b94674b1fde17e4e460006" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } peers { permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 38525 } attrs { promote: false } } peers { permanent_uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 43695 } attrs { promote: false } } }
I20260924 11:34:35.677549 13997 sys_catalog.cc:458] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 [sys.catalog]: This master's current role is: LEADER
I20260924 11:34:35.684425 14143 sys_catalog.cc:455] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f32c56b9b2b94674b1fde17e4e460006" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } peers { permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 38525 } attrs { promote: false } } peers { permanent_uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 43695 } attrs { promote: false } } }
I20260924 11:34:35.684918 14143 sys_catalog.cc:458] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:34:35.687812 14137 sys_catalog.cc:455] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f32c56b9b2b94674b1fde17e4e460006" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } peers { permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 38525 } attrs { promote: false } } peers { permanent_uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 43695 } attrs { promote: false } } }
I20260924 11:34:35.688153 14137 sys_catalog.cc:458] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006 [sys.catalog]: This master's current role is: LEADER
I20260924 11:34:35.688980 13953 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bbe9da2f8b0e4ed79c32eb5d554a06e5 (127.6.212.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } peers { permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 38525 } attrs { promote: false } } peers { permanent_uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 43695 } attrs { promote: false } } }
I20260924 11:34:35.695875 14144 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f32c56b9b2b94674b1fde17e4e460006" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } peers { permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 38525 } attrs { promote: false } } peers { permanent_uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 43695 } attrs { promote: false } } }
I20260924 11:34:35.696457 14144 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 11:34:35.750304 14136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:35.750676 14136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:35.750773 14136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:35.758415 14136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:35.758553 14136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:35.758667 14136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.2
I20260924 11:34:35.769276 14136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:33661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.2.keytab
--principal=oryx/127.6.212.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.2:34887
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.212.2
--webserver_port=34395
--tserver_master_addrs=127.6.212.62:42625,127.6.212.61:38525,127.6.212.60:43695
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.2
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:34:35.770917 14136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:35.772302 14136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:35.787635 14152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:35.787853 14150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:35.787891 14149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:35.788637 14136 server_base.cc:1034] running on GCE node
I20260924 11:34:35.789366 14136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:35.790428 14136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:35.791678 14136 hybrid_clock.cc:648] HybridClock initialized: now 1790249675791615 us; error 55 us; skew 500 ppm
I20260924 11:34:35.791973 14136 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:34:35 dist-test-slave-cq7c krb5kdc[13114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249675, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.212.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:35.795218 14136 init.cc:377] Logged in from keytab as oryx/127.6.212.2@KRBTEST.COM (short username oryx)
I20260924 11:34:35.796869 14136 webserver.cc:460] Webserver started at http://127.6.212.2:34395/ using document root <none> and password file <none>
I20260924 11:34:35.797646 14136 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:35.797766 14136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:35.803581 14136 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 11:34:35.806517 14159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:35.808393 14136 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:34:35.808583 14136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-1/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-1/wal
uuid: "faaacac03e5747f48f17a2b84561a5d8"
format_stamp: "Formatted at 2026-09-24 11:34:32 on dist-test-slave-cq7c"
server_key: "06327aa6db175fea2a4472d47c692551"
server_key_iv: "a107241b67b5984fb7cf64bf51abe268"
server_key_version: "encryptionkey@0"
I20260924 11:34:35.809597 14136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:35.861516 14136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:35.866130 14136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:34:35.866407 14136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:35.867513 14136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:34:35.868907 14136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:34:35.869010 14136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:35.869098 14136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:34:35.869161 14136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:35.903939 14136 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.2:34887
I20260924 11:34:35.904037 14271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.2:34887 every 8 connection(s)
I20260924 11:34:35.905473 14136 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
I20260924 11:34:35.913741 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 14136
I20260924 11:34:35.914022 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 13487
Sep 24 11:34:35 dist-test-slave-cq7c krb5kdc[13114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.212.3@KRBTEST.COM
I20260924 11:34:35.949513 14283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:35.915148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54940 (local address 127.6.212.3:45033)
0924 11:34:35.918911 (+ 3763us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:35.918935 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:35.918976 (+ 41us) server_negotiation.cc:407] Connection header received
0924 11:34:35.919069 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:35.919082 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:35.919468 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:35.919666 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:35.924137 (+ 4471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:35.926423 (+ 2286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:35.931509 (+ 5086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:35.932511 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:35.932648 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:35.940113 (+ 7465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:35.940134 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:35.940154 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:35.940202 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:35.945050 (+ 4848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:35.945985 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:35.945997 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:35.946001 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:35.946163 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:35.946665 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:35.946670 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:35.946673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:35.947124 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:35.947484 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:35.948646 (+ 1162us) server_negotiation.cc:299] Negotiation successful
0924 11:34:35.948946 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3505,"thread_start_us":1933,"threads_started":1}
Sep 24 11:34:35 dist-test-slave-cq7c krb5kdc[13114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249675, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.212.2@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM
Sep 24 11:34:35 dist-test-slave-cq7c krb5kdc[13114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249675, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.212.2@KRBTEST.COM for oryx/127.6.212.60@KRBTEST.COM
Sep 24 11:34:35 dist-test-slave-cq7c krb5kdc[13114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249675, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.212.2@KRBTEST.COM for oryx/127.6.212.61@KRBTEST.COM
I20260924 11:34:35.952096 13610 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:34:35.960355 14277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:35.909006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:59349 (local address 127.6.212.60:43695)
0924 11:34:35.935228 (+ 26222us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:35.935235 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:35.937111 (+ 1876us) server_negotiation.cc:407] Connection header received
0924 11:34:35.937183 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:35.937190 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:35.937283 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:35.937389 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:35.939962 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:35.941171 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:35.944808 (+ 3637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:35.945490 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:35.945597 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:35.952009 (+ 6412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:35.952018 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:35.952023 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:35.952071 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:35.955817 (+ 3746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:35.957142 (+ 1325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:35.957150 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:35.957156 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:35.957299 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:35.957865 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:35.957870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:35.957874 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:35.958232 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:35.958370 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:35.959719 (+ 1349us) server_negotiation.cc:299] Negotiation successful
0924 11:34:35.959969 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26075,"thread_start_us":25941,"threads_started":1}
I20260924 11:34:35.961355 14279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:35.909730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:51169 (local address 127.6.212.61:38525)
0924 11:34:35.937855 (+ 28125us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:35.937865 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:35.937884 (+ 19us) server_negotiation.cc:407] Connection header received
0924 11:34:35.937951 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:35.937957 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:35.938048 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:35.938164 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:35.940223 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:35.941287 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:35.944787 (+ 3500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:35.945441 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:35.945562 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:35.952540 (+ 6978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:35.952557 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:35.952563 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:35.952622 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:35.956243 (+ 3621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:35.957184 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:35.957189 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:35.957194 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:35.957321 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:35.957865 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:35.957870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:35.957874 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:35.958235 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:35.958378 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:35.960815 (+ 2437us) server_negotiation.cc:299] Negotiation successful
0924 11:34:35.961029 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27980,"thread_start_us":27816,"threads_started":1}
I20260924 11:34:35.961593 14276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:35.908767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:35259 (local address 127.6.212.62:42625)
0924 11:34:35.931403 (+ 22636us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:35.931411 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:35.934791 (+ 3380us) server_negotiation.cc:407] Connection header received
0924 11:34:35.936357 (+ 1566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:35.936366 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:35.936491 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:35.936584 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:35.940055 (+ 3471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:35.941205 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:35.944787 (+ 3582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:35.945441 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:35.945546 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:35.951809 (+ 6263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:35.951816 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:35.951821 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:35.951877 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:35.957487 (+ 5610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:35.958621 (+ 1134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:35.958630 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:35.958637 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:35.958841 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:35.959432 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:35.959440 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:35.959446 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:35.959918 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:35.960103 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:35.960710 (+ 607us) server_negotiation.cc:299] Negotiation successful
0924 11:34:35.961055 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22457,"thread_start_us":22605,"threads_started":1}
I20260924 11:34:35.962163 14281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:35.909973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:43695 (local address 127.6.212.2:59349)
0924 11:34:35.936766 (+ 26793us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:35.936803 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:35.936977 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:35.937721 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:35.937726 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:35.938562 (+ 836us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:35.939742 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:35.939756 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:35.941417 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:35.941421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:35.944626 (+ 3205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:35.944640 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:35.945895 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:35.945903 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:35.946370 (+ 467us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:35.947673 (+ 1303us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:35.947697 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:35.951748 (+ 4051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:35.956027 (+ 4279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:35.956039 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:35.956060 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:35.956978 (+ 918us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:35.957437 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:35.957442 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:35.957446 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:35.957722 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:35.958444 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:35.958458 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:35.959023 (+ 565us) client_negotiation.cc:769] Sending connection context
0924 11:34:35.959402 (+ 379us) client_negotiation.cc:240] Negotiation successful
0924 11:34:35.960927 (+ 1525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26640,"mutex_wait_us":1736,"spinlock_wait_cycles":52992,"thread_start_us":29070,"threads_started":1}
I20260924 11:34:35.962198 14280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:35.909463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:38525 (local address 127.6.212.2:51169)
0924 11:34:35.937212 (+ 27749us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:35.937230 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:35.937332 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:35.938519 (+ 1187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:35.938524 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:35.939755 (+ 1231us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:35.940043 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:35.940055 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:35.941438 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:35.941455 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:35.944627 (+ 3172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:35.944636 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:35.945879 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:35.945887 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:35.946367 (+ 480us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:35.947342 (+ 975us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:35.947394 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:35.952248 (+ 4854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:35.956407 (+ 4159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:35.956413 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:35.956416 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:35.957052 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:35.957437 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:35.957455 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:35.957457 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:35.957720 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:35.958458 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:35.958462 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:35.959023 (+ 561us) client_negotiation.cc:769] Sending connection context
0924 11:34:35.959402 (+ 379us) client_negotiation.cc:240] Negotiation successful
0924 11:34:35.959979 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27598,"mutex_wait_us":1686,"spinlock_wait_cycles":33408,"thread_start_us":25957,"threads_started":1}
I20260924 11:34:35.963163 14278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:35.909463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.6.212.2:35259)
0924 11:34:35.932281 (+ 22818us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:35.932341 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:35.935880 (+ 3539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:35.937314 (+ 1434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:35.937329 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:35.938299 (+ 970us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:35.939663 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:35.939692 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:35.941416 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:35.941421 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:35.944594 (+ 3173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:35.944611 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:35.945880 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:35.945887 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:35.946397 (+ 510us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:35.947391 (+ 994us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:35.947412 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:35.951504 (+ 4092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:35.957732 (+ 6228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:35.957740 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:35.957746 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:35.958423 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:35.959028 (+ 605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:35.959035 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:35.959041 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:35.959253 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:35.960207 (+ 954us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:35.960219 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:35.960432 (+ 213us) client_negotiation.cc:769] Sending connection context
0924 11:34:35.961845 (+ 1413us) client_negotiation.cc:240] Negotiation successful
0924 11:34:35.962290 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22339,"mutex_wait_us":559,"thread_start_us":22114,"threads_started":1}
I20260924 11:34:35.963963 14272 heartbeater.cc:347] Connected to a master server at 127.6.212.60:43695
I20260924 11:34:35.964032 14273 heartbeater.cc:347] Connected to a master server at 127.6.212.62:42625
I20260924 11:34:35.964316 14272 heartbeater.cc:464] Registering TS with master...
I20260924 11:34:35.964329 14273 heartbeater.cc:464] Registering TS with master...
I20260924 11:34:35.965082 14274 heartbeater.cc:347] Connected to a master server at 127.6.212.61:38525
I20260924 11:34:35.965216 14274 heartbeater.cc:464] Registering TS with master...
I20260924 11:34:35.965595 14273 heartbeater.cc:511] Master 127.6.212.62:42625 requested a full tablet report, sending...
I20260924 11:34:35.965651 14274 heartbeater.cc:511] Master 127.6.212.61:38525 requested a full tablet report, sending...
I20260924 11:34:35.966014 14272 heartbeater.cc:511] Master 127.6.212.60:43695 requested a full tablet report, sending...
I20260924 11:34:35.967844 13757 ts_manager.cc:194] Registered new tserver with Master: faaacac03e5747f48f17a2b84561a5d8 (127.6.212.2:34887)
I20260924 11:34:35.969039 13158 ts_manager.cc:194] Re-registered known tserver with Master: faaacac03e5747f48f17a2b84561a5d8 (127.6.212.2:34887)
I20260924 11:34:35.970366 13158 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.212.2@KRBTEST.COM'} at 127.6.212.2:35259
I20260924 11:34:35.973481 13943 ts_manager.cc:194] Registered new tserver with Master: faaacac03e5747f48f17a2b84561a5d8 (127.6.212.2:34887)
I20260924 11:34:36.052711 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 13487
I20260924 11:34:36.070505 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.3:45033
--local_ip_for_outbound_sockets=127.6.212.3
--tserver_master_addrs=127.6.212.62:42625
--webserver_port=45459
--webserver_interface=127.6.212.3
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.3.keytab
--principal=oryx/127.6.212.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:33661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.212.62:42625,127.6.212.61:38525
--tserver_master_addrs=127.6.212.62:42625,127.6.212.61:38525,127.6.212.60:43695 with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:34:36.290743 14125 heartbeater.cc:503] Master 127.6.212.62:42625 was elected leader, sending a full tablet report...
W20260924 11:34:36.415462 14285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:36.415766 14285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:36.415860 14285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:36.423111 14285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:36.423223 14285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:36.423337 14285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.3
I20260924 11:34:36.433212 14285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:33661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.3.keytab
--principal=oryx/127.6.212.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.3:45033
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.212.3
--webserver_port=45459
--tserver_master_addrs=127.6.212.62:42625,127.6.212.61:38525,127.6.212.60:43695
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.3
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:34:36.434557 14285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:36.436070 14285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:36.449792 14293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:36.450228 14291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:36.449744 14290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:36.457176 14285 server_base.cc:1034] running on GCE node
I20260924 11:34:36.457919 14285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:36.458875 14285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:36.460081 14285 hybrid_clock.cc:648] HybridClock initialized: now 1790249676460035 us; error 37 us; skew 500 ppm
I20260924 11:34:36.460374 14285 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:34:36 dist-test-slave-cq7c krb5kdc[13114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249676, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.212.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:36.464129 14285 init.cc:377] Logged in from keytab as oryx/127.6.212.3@KRBTEST.COM (short username oryx)
I20260924 11:34:36.465750 14285 webserver.cc:460] Webserver started at http://127.6.212.3:45459/ using document root <none> and password file <none>
I20260924 11:34:36.466554 14285 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:36.466719 14285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:36.472728 14285 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 11:34:36.475533 14300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:36.476956 14285 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 11:34:36.477136 14285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-2/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-2/wal
uuid: "b1b300a0baf545cebef745d22579f93e"
format_stamp: "Formatted at 2026-09-24 11:34:32 on dist-test-slave-cq7c"
server_key: "04b674aa114a84e423fc153658b05d11"
server_key_iv: "e391d6924008b9ec08bc36a7b6e01195"
server_key_version: "encryptionkey@0"
I20260924 11:34:36.478130 14285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:36.498602 14285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:36.502817 14285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:34:36.503101 14285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:36.504135 14285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:34:36.505509 14285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:34:36.505628 14285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:36.505766 14285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:34:36.505856 14285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:36.541553 14285 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.3:45033
I20260924 11:34:36.541618 14412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.3:45033 every 8 connection(s)
I20260924 11:34:36.543021 14285 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
I20260924 11:34:36.552734 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 14285
Sep 24 11:34:36 dist-test-slave-cq7c krb5kdc[13114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249676, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.212.3@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM
Sep 24 11:34:36 dist-test-slave-cq7c krb5kdc[13114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249676, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.212.3@KRBTEST.COM for oryx/127.6.212.60@KRBTEST.COM
I20260924 11:34:36.592722 14422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:36.546668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.6.212.3:55103)
0924 11:34:36.556067 (+ 9399us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:36.556134 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:36.565247 (+ 9113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:36.566054 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:36.566071 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:36.570082 (+ 4011us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:36.570437 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:36.570454 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:36.572610 (+ 2156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:36.572618 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:36.574907 (+ 2289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:36.574929 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:36.576972 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:36.576981 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:36.577262 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:36.578025 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:36.578059 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:36.583386 (+ 5327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:36.587515 (+ 4129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:36.587525 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:36.587546 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:36.588196 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:36.588660 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:36.588665 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:36.588668 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:36.588932 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:36.589663 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:36.589672 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:36.590139 (+ 467us) client_negotiation.cc:769] Sending connection context
0924 11:34:36.590443 (+ 304us) client_negotiation.cc:240] Negotiation successful
0924 11:34:36.591023 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9046,"mutex_wait_us":1,"thread_start_us":26917,"threads_started":1}
I20260924 11:34:36.594295 14417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:36.545941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:55103 (local address 127.6.212.62:42625)
0924 11:34:36.550090 (+ 4149us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:36.550096 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:36.556281 (+ 6185us) server_negotiation.cc:407] Connection header received
0924 11:34:36.565416 (+ 9135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:36.565425 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:36.565546 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:36.565720 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:36.570694 (+ 4974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:36.572336 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:36.575517 (+ 3181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:36.576462 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:36.576591 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:36.583623 (+ 7032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:36.583630 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:36.583637 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:36.583671 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:36.587302 (+ 3631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:36.588340 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:36.588345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:36.588349 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:36.588515 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:36.589072 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:36.589076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:36.589081 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:36.589437 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:36.589589 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:36.593686 (+ 4097us) server_negotiation.cc:299] Negotiation successful
0924 11:34:36.593951 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4011,"thread_start_us":3908,"threads_started":1}
I20260924 11:34:36.595770 14414 heartbeater.cc:347] Connected to a master server at 127.6.212.62:42625
I20260924 11:34:36.596122 14414 heartbeater.cc:464] Registering TS with master...
I20260924 11:34:36.597486 14414 heartbeater.cc:511] Master 127.6.212.62:42625 requested a full tablet report, sending...
Sep 24 11:34:36 dist-test-slave-cq7c krb5kdc[13114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249676, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.212.3@KRBTEST.COM for oryx/127.6.212.61@KRBTEST.COM
I20260924 11:34:36.599701 13158 ts_manager.cc:194] Re-registered known tserver with Master: b1b300a0baf545cebef745d22579f93e (127.6.212.3:45033)
I20260924 11:34:36.600904 13158 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.212.3@KRBTEST.COM'} at 127.6.212.3:55103
I20260924 11:34:36.606139 14420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:36.546976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:53331 (local address 127.6.212.60:43695)
0924 11:34:36.564490 (+ 17514us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:36.564499 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:36.564529 (+ 30us) server_negotiation.cc:407] Connection header received
0924 11:34:36.565423 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:36.565432 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:36.565552 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:36.569315 (+ 3763us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:36.569519 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:36.570756 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:36.575004 (+ 4248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:36.575954 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:36.576110 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:36.588468 (+ 12358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:36.588478 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:36.588486 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:36.588540 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:36.592874 (+ 4334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:36.594355 (+ 1481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:36.594360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:36.594364 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:36.594542 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:36.595009 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:36.595017 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:36.595023 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:36.599633 (+ 4610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:36.599825 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:36.604937 (+ 5112us) server_negotiation.cc:299] Negotiation successful
0924 11:34:36.605267 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17348,"thread_start_us":17556,"threads_started":1}
I20260924 11:34:36.608482 14419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:36.546669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:38525 (local address 127.6.212.3:36123)
0924 11:34:36.574098 (+ 27429us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:36.574124 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:36.574278 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:36.578207 (+ 3929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:36.578214 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:36.581470 (+ 3256us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:36.581829 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:36.581848 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:36.591611 (+ 9763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:36.591622 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:36.593683 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:36.593699 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:36.594763 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:36.594772 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:36.594952 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:36.595569 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:36.595585 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:36.598273 (+ 2688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:36.604916 (+ 6643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:36.604921 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:36.604925 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:36.605442 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:36.606653 (+ 1211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:36.606659 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:36.606663 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:36.606816 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:36.607479 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:36.607483 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:36.607654 (+ 171us) client_negotiation.cc:769] Sending connection context
0924 11:34:36.607774 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 11:34:36.608091 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27167,"mutex_wait_us":922,"thread_start_us":7969,"threads_started":1}
I20260924 11:34:36.609714 14418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:36.546000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:36123 (local address 127.6.212.61:38525)
0924 11:34:36.577818 (+ 31818us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:36.577825 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:36.577846 (+ 21us) server_negotiation.cc:407] Connection header received
0924 11:34:36.577924 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:36.577930 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:36.578018 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:36.578139 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:36.589289 (+ 11150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:36.591472 (+ 2183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:36.593820 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:36.594572 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:36.594699 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:36.598489 (+ 3790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:36.598505 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:36.598510 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:36.598563 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:36.604757 (+ 6194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:36.605618 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:36.605624 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:36.605629 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:36.606506 (+ 877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:36.606957 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:36.606962 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:36.606966 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:36.607306 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:36.608955 (+ 1649us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:36.609093 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 11:34:36.609323 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31672,"thread_start_us":31878,"threads_started":1}
I20260924 11:34:36.610733 14415 heartbeater.cc:347] Connected to a master server at 127.6.212.61:38525
I20260924 11:34:36.610895 14415 heartbeater.cc:464] Registering TS with master...
I20260924 11:34:36.611465 14415 heartbeater.cc:511] Master 127.6.212.61:38525 requested a full tablet report, sending...
I20260924 11:34:36.612874 13757 ts_manager.cc:194] Registered new tserver with Master: b1b300a0baf545cebef745d22579f93e (127.6.212.3:45033)
I20260924 11:34:36.612937 14421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:36.546668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:43695 (local address 127.6.212.3:53331)
0924 11:34:36.559883 (+ 13215us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:36.559921 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:36.564756 (+ 4835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:36.567006 (+ 2250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:36.567014 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:36.567898 (+ 884us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:36.569085 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:36.569117 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:36.570986 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:36.570994 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:36.574755 (+ 3761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:36.574781 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:36.576518 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:36.576528 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:36.576966 (+ 438us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:36.578117 (+ 1151us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:36.578139 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:36.587346 (+ 9207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:36.593637 (+ 6291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:36.593645 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:36.593651 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:36.594207 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:36.594681 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:36.594686 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:36.594689 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:36.594852 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:36.599930 (+ 5078us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:36.599938 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:36.600140 (+ 202us) client_negotiation.cc:769] Sending connection context
0924 11:34:36.600319 (+ 179us) client_negotiation.cc:240] Negotiation successful
0924 11:34:36.600735 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13030,"mutex_wait_us":1464,"thread_start_us":12244,"threads_started":1}
I20260924 11:34:36.615660 14413 heartbeater.cc:347] Connected to a master server at 127.6.212.60:43695
I20260924 11:34:36.615788 14413 heartbeater.cc:464] Registering TS with master...
I20260924 11:34:36.616201 14413 heartbeater.cc:511] Master 127.6.212.60:43695 requested a full tablet report, sending...
I20260924 11:34:36.617735 13943 ts_manager.cc:194] Registered new tserver with Master: b1b300a0baf545cebef745d22579f93e (127.6.212.3:45033)
I20260924 11:34:36.618557 14430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:36.581069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50098 (local address 127.6.212.62:42625)
0924 11:34:36.583051 (+ 1982us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:36.583056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:36.583076 (+ 20us) server_negotiation.cc:407] Connection header received
0924 11:34:36.583135 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:36.583142 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:36.583227 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:36.583334 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:36.586615 (+ 3281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:36.587897 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:36.590728 (+ 2831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:36.591318 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:36.591470 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:36.609286 (+ 17816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:36.609292 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:36.609297 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:36.609334 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:36.614541 (+ 5207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:36.615219 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:36.615227 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:36.615233 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:36.615409 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:36.616031 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:36.616035 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:36.616038 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:36.616552 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:36.616772 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:36.618025 (+ 1253us) server_negotiation.cc:299] Negotiation successful
0924 11:34:36.618245 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1837,"thread_start_us":2479,"threads_started":1}
I20260924 11:34:36.626794 14427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:36.580540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43654 (local address 127.6.212.61:38525)
0924 11:34:36.588394 (+ 7854us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:36.588400 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:36.597585 (+ 9185us) server_negotiation.cc:407] Connection header received
0924 11:34:36.599654 (+ 2069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:36.599660 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:36.599742 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:36.599820 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:36.605699 (+ 5879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:36.607334 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:36.615347 (+ 8013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:36.616048 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:36.616191 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:36.620541 (+ 4350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:36.620551 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:36.620557 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:36.620596 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:36.623938 (+ 3342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:36.624761 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:36.624767 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:36.624772 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:36.624920 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:36.625377 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:36.625382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:36.625385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:36.625661 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:36.625897 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:36.626293 (+ 396us) server_negotiation.cc:299] Negotiation successful
0924 11:34:36.626500 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7689,"thread_start_us":7611,"threads_started":1}
I20260924 11:34:36.628719 14420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:36.581287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52910 (local address 127.6.212.60:43695)
0924 11:34:36.606467 (+ 25180us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:36.606475 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:36.606499 (+ 24us) server_negotiation.cc:407] Connection header received
0924 11:34:36.606563 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:36.606570 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:36.606708 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:36.610787 (+ 4079us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:36.612158 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:36.614564 (+ 2406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:36.616962 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:36.617708 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:36.618889 (+ 1181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:36.621365 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:36.621394 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:36.621401 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:36.621444 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:36.624932 (+ 3488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:36.625806 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:36.625813 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:36.625817 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:36.625962 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:36.626363 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:36.626368 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:36.626371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:36.626712 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:36.626939 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:36.628121 (+ 1182us) server_negotiation.cc:299] Negotiation successful
0924 11:34:36.628353 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25019,"thread_start_us":33567,"threads_started":1}
I20260924 11:34:36.623769 13158 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50098:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:34:36.632666 13158 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:34:36.671456 14441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:36.652309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:42317 (local address 127.0.0.1:46044)
0924 11:34:36.659573 (+ 7264us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:36.659600 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:36.659773 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:36.661102 (+ 1329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:36.661110 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:36.661146 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:34:36.661709 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:36.661728 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:36.665182 (+ 3454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:36.665189 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:36.668080 (+ 2891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:36.668094 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:36.670421 (+ 2327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:36.670428 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:36.670695 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:36.670739 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 11:34:36.670818 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 11:34:36.670958 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":794,"mutex_wait_us":23,"thread_start_us":350,"threads_started":1}
I20260924 11:34:36.671972 14442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:36.652541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46044 (local address 127.6.212.1:42317)
0924 11:34:36.659209 (+ 6668us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:36.659220 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:36.659947 (+ 727us) server_negotiation.cc:407] Connection header received
0924 11:34:36.660095 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:36.660108 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:36.660530 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:36.660797 (+ 267us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:34:36.661970 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:36.665024 (+ 3054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:36.668330 (+ 3306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:36.669943 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:36.670072 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:36.670301 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:36.671501 (+ 1200us) server_negotiation.cc:299] Negotiation successful
0924 11:34:36.671675 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6457,"thread_start_us":6695,"threads_started":1}
I20260924 11:34:36.672076 14445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:36.653061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.3:45033 (local address 127.0.0.1:41866)
0924 11:34:36.664124 (+ 11063us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:36.664165 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:36.664296 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:36.665300 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:36.665307 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:36.665337 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:34:36.666225 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:36.666241 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:36.667762 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:36.667774 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:36.669380 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:36.669392 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:36.671197 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:36.671203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:36.671414 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:36.671456 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 11:34:36.671537 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 11:34:36.671668 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10928,"mutex_wait_us":392,"spinlock_wait_cycles":11392,"thread_start_us":10187,"threads_started":1}
I20260924 11:34:36.673660 14446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:36.653740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41866 (local address 127.6.212.3:45033)
0924 11:34:36.663879 (+ 10139us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:36.663889 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:36.664690 (+ 801us) server_negotiation.cc:407] Connection header received
0924 11:34:36.664780 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:36.664786 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:36.665011 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:36.665325 (+ 314us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:34:36.666460 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:36.667624 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:36.669535 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:36.670979 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:36.671121 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:36.671333 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:36.673077 (+ 1744us) server_negotiation.cc:299] Negotiation successful
0924 11:34:36.673272 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9927,"thread_start_us":9641,"threads_started":1}
I20260924 11:34:36.673748 14444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:36.653276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42092 (local address 127.6.212.2:34887)
0924 11:34:36.664748 (+ 11472us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:36.664757 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:36.664784 (+ 27us) server_negotiation.cc:407] Connection header received
0924 11:34:36.664868 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:36.664875 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:36.665283 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:36.665923 (+ 640us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:34:36.666850 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:36.668361 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:36.670875 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:36.672737 (+ 1862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:36.672839 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:36.673006 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:36.673322 (+ 316us) server_negotiation.cc:299] Negotiation successful
0924 11:34:36.673508 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11321,"thread_start_us":11226,"threads_started":1}
I20260924 11:34:36.673872 14443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:36.653016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.2:34887 (local address 127.0.0.1:42092)
0924 11:34:36.653994 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:36.654017 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:36.654155 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:36.665936 (+ 11781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:36.665947 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:36.665988 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:34:36.666648 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:36.666661 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:36.668499 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:36.668505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:36.670676 (+ 2171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:36.670689 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:36.672951 (+ 2262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:36.672959 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:36.673130 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:36.673161 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 11:34:36.673264 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 11:34:36.673405 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":838,"mutex_wait_us":179,"spinlock_wait_cycles":768,"thread_start_us":704,"threads_started":1}
I20260924 11:34:36.676271 14059 tablet_service.cc:1467] Processing CreateTablet for tablet 5ea281391bc24507a335f41a18079c0c (DEFAULT_TABLE table=test-table [id=1cb515a8b01b4250af3c53f00fc863e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:34:36.677841 14059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ea281391bc24507a335f41a18079c0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:34:36.677325 14348 tablet_service.cc:1467] Processing CreateTablet for tablet 5ea281391bc24507a335f41a18079c0c (DEFAULT_TABLE table=test-table [id=1cb515a8b01b4250af3c53f00fc863e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:34:36.678598 14348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ea281391bc24507a335f41a18079c0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:34:36.677429 14207 tablet_service.cc:1467] Processing CreateTablet for tablet 5ea281391bc24507a335f41a18079c0c (DEFAULT_TABLE table=test-table [id=1cb515a8b01b4250af3c53f00fc863e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:34:36.679725 14207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ea281391bc24507a335f41a18079c0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:34:36.700827 14448 tablet_bootstrap.cc:492] T 5ea281391bc24507a335f41a18079c0c P b1b300a0baf545cebef745d22579f93e: Bootstrap starting.
I20260924 11:34:36.701306 14447 tablet_bootstrap.cc:492] T 5ea281391bc24507a335f41a18079c0c P 92c42ff207f3406aa06f3984a03fc08b: Bootstrap starting.
I20260924 11:34:36.705384 14448 tablet_bootstrap.cc:654] T 5ea281391bc24507a335f41a18079c0c P b1b300a0baf545cebef745d22579f93e: Neither blocks nor log segments found. Creating new log.
I20260924 11:34:36.705384 14447 tablet_bootstrap.cc:654] T 5ea281391bc24507a335f41a18079c0c P 92c42ff207f3406aa06f3984a03fc08b: Neither blocks nor log segments found. Creating new log.
I20260924 11:34:36.705713 14449 tablet_bootstrap.cc:492] T 5ea281391bc24507a335f41a18079c0c P faaacac03e5747f48f17a2b84561a5d8: Bootstrap starting.
I20260924 11:34:36.706743 14448 log.cc:824] T 5ea281391bc24507a335f41a18079c0c P b1b300a0baf545cebef745d22579f93e: Log is configured to *not* fsync() on all Append() calls
I20260924 11:34:36.707265 14447 log.cc:824] T 5ea281391bc24507a335f41a18079c0c P 92c42ff207f3406aa06f3984a03fc08b: Log is configured to *not* fsync() on all Append() calls
I20260924 11:34:36.709622 14449 tablet_bootstrap.cc:654] T 5ea281391bc24507a335f41a18079c0c P faaacac03e5747f48f17a2b84561a5d8: Neither blocks nor log segments found. Creating new log.
I20260924 11:34:36.710659 14449 log.cc:824] T 5ea281391bc24507a335f41a18079c0c P faaacac03e5747f48f17a2b84561a5d8: Log is configured to *not* fsync() on all Append() calls
I20260924 11:34:36.710765 14448 tablet_bootstrap.cc:492] T 5ea281391bc24507a335f41a18079c0c P b1b300a0baf545cebef745d22579f93e: No bootstrap required, opened a new log
I20260924 11:34:36.711054 14448 ts_tablet_manager.cc:1397] T 5ea281391bc24507a335f41a18079c0c P b1b300a0baf545cebef745d22579f93e: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260924 11:34:36.711642 14447 tablet_bootstrap.cc:492] T 5ea281391bc24507a335f41a18079c0c P 92c42ff207f3406aa06f3984a03fc08b: No bootstrap required, opened a new log
I20260924 11:34:36.711987 14447 ts_tablet_manager.cc:1397] T 5ea281391bc24507a335f41a18079c0c P 92c42ff207f3406aa06f3984a03fc08b: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260924 11:34:36.713363 14449 tablet_bootstrap.cc:492] T 5ea281391bc24507a335f41a18079c0c P faaacac03e5747f48f17a2b84561a5d8: No bootstrap required, opened a new log
I20260924 11:34:36.713608 14449 ts_tablet_manager.cc:1397] T 5ea281391bc24507a335f41a18079c0c P faaacac03e5747f48f17a2b84561a5d8: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 11:34:36.722119 14449 raft_consensus.cc:348] T 5ea281391bc24507a335f41a18079c0c P faaacac03e5747f48f17a2b84561a5d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c42ff207f3406aa06f3984a03fc08b" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42317 } } peers { permanent_uuid: "faaacac03e5747f48f17a2b84561a5d8" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 34887 } } peers { permanent_uuid: "b1b300a0baf545cebef745d22579f93e" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 45033 } }
I20260924 11:34:36.722178 14448 raft_consensus.cc:348] T 5ea281391bc24507a335f41a18079c0c P b1b300a0baf545cebef745d22579f93e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c42ff207f3406aa06f3984a03fc08b" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42317 } } peers { permanent_uuid: "faaacac03e5747f48f17a2b84561a5d8" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 34887 } } peers { permanent_uuid: "b1b300a0baf545cebef745d22579f93e" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 45033 } }
I20260924 11:34:36.722550 14449 raft_consensus.cc:374] T 5ea281391bc24507a335f41a18079c0c P faaacac03e5747f48f17a2b84561a5d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:34:36.722640 14448 raft_consensus.cc:374] T 5ea281391bc24507a335f41a18079c0c P b1b300a0baf545cebef745d22579f93e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:34:36.722637 14449 raft_consensus.cc:729] T 5ea281391bc24507a335f41a18079c0c P faaacac03e5747f48f17a2b84561a5d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faaacac03e5747f48f17a2b84561a5d8, State: Initialized, Role: FOLLOWER
I20260924 11:34:36.722749 14448 raft_consensus.cc:729] T 5ea281391bc24507a335f41a18079c0c P b1b300a0baf545cebef745d22579f93e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1b300a0baf545cebef745d22579f93e, State: Initialized, Role: FOLLOWER
I20260924 11:34:36.723315 14449 consensus_queue.cc:260] T 5ea281391bc24507a335f41a18079c0c P faaacac03e5747f48f17a2b84561a5d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c42ff207f3406aa06f3984a03fc08b" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42317 } } peers { permanent_uuid: "faaacac03e5747f48f17a2b84561a5d8" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 34887 } } peers { permanent_uuid: "b1b300a0baf545cebef745d22579f93e" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 45033 } }
I20260924 11:34:36.723107 14447 raft_consensus.cc:348] T 5ea281391bc24507a335f41a18079c0c P 92c42ff207f3406aa06f3984a03fc08b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c42ff207f3406aa06f3984a03fc08b" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42317 } } peers { permanent_uuid: "faaacac03e5747f48f17a2b84561a5d8" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 34887 } } peers { permanent_uuid: "b1b300a0baf545cebef745d22579f93e" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 45033 } }
I20260924 11:34:36.723987 14447 raft_consensus.cc:374] T 5ea281391bc24507a335f41a18079c0c P 92c42ff207f3406aa06f3984a03fc08b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:34:36.723860 14448 consensus_queue.cc:260] T 5ea281391bc24507a335f41a18079c0c P b1b300a0baf545cebef745d22579f93e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c42ff207f3406aa06f3984a03fc08b" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42317 } } peers { permanent_uuid: "faaacac03e5747f48f17a2b84561a5d8" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 34887 } } peers { permanent_uuid: "b1b300a0baf545cebef745d22579f93e" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 45033 } }
I20260924 11:34:36.724095 14447 raft_consensus.cc:729] T 5ea281391bc24507a335f41a18079c0c P 92c42ff207f3406aa06f3984a03fc08b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92c42ff207f3406aa06f3984a03fc08b, State: Initialized, Role: FOLLOWER
I20260924 11:34:36.724712 14273 heartbeater.cc:503] Master 127.6.212.62:42625 was elected leader, sending a full tablet report...
I20260924 11:34:36.725193 14449 ts_tablet_manager.cc:1428] T 5ea281391bc24507a335f41a18079c0c P faaacac03e5747f48f17a2b84561a5d8: Time spent starting tablet: real 0.011s user 0.008s sys 0.005s
I20260924 11:34:36.728180 14447 consensus_queue.cc:260] T 5ea281391bc24507a335f41a18079c0c P 92c42ff207f3406aa06f3984a03fc08b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c42ff207f3406aa06f3984a03fc08b" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42317 } } peers { permanent_uuid: "faaacac03e5747f48f17a2b84561a5d8" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 34887 } } peers { permanent_uuid: "b1b300a0baf545cebef745d22579f93e" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 45033 } }
I20260924 11:34:36.732502 14447 ts_tablet_manager.cc:1428] T 5ea281391bc24507a335f41a18079c0c P 92c42ff207f3406aa06f3984a03fc08b: Time spent starting tablet: real 0.020s user 0.014s sys 0.000s
I20260924 11:34:36.733598 14414 heartbeater.cc:503] Master 127.6.212.62:42625 was elected leader, sending a full tablet report...
I20260924 11:34:36.736347 14448 ts_tablet_manager.cc:1428] T 5ea281391bc24507a335f41a18079c0c P b1b300a0baf545cebef745d22579f93e: Time spent starting tablet: real 0.025s user 0.016s sys 0.000s
W20260924 11:34:36.751449 14127 tablet.cc:2369] T 5ea281391bc24507a335f41a18079c0c P 92c42ff207f3406aa06f3984a03fc08b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:34:36.820622 14416 tablet.cc:2369] T 5ea281391bc24507a335f41a18079c0c P b1b300a0baf545cebef745d22579f93e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:34:36.884994 14453 raft_consensus.cc:482] T 5ea281391bc24507a335f41a18079c0c P faaacac03e5747f48f17a2b84561a5d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:34:36.885423 14453 raft_consensus.cc:504] T 5ea281391bc24507a335f41a18079c0c P faaacac03e5747f48f17a2b84561a5d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c42ff207f3406aa06f3984a03fc08b" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42317 } } peers { permanent_uuid: "faaacac03e5747f48f17a2b84561a5d8" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 34887 } } peers { permanent_uuid: "b1b300a0baf545cebef745d22579f93e" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 45033 } }
I20260924 11:34:36.887207 14453 leader_election.cc:288] T 5ea281391bc24507a335f41a18079c0c P faaacac03e5747f48f17a2b84561a5d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92c42ff207f3406aa06f3984a03fc08b (127.6.212.1:42317), b1b300a0baf545cebef745d22579f93e (127.6.212.3:45033)
I20260924 11:34:36.888012 14454 raft_consensus.cc:482] T 5ea281391bc24507a335f41a18079c0c P b1b300a0baf545cebef745d22579f93e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:34:36.888326 14454 raft_consensus.cc:504] T 5ea281391bc24507a335f41a18079c0c P b1b300a0baf545cebef745d22579f93e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c42ff207f3406aa06f3984a03fc08b" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42317 } } peers { permanent_uuid: "faaacac03e5747f48f17a2b84561a5d8" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 34887 } } peers { permanent_uuid: "b1b300a0baf545cebef745d22579f93e" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 45033 } }
I20260924 11:34:36.890352 14454 leader_election.cc:288] T 5ea281391bc24507a335f41a18079c0c P b1b300a0baf545cebef745d22579f93e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92c42ff207f3406aa06f3984a03fc08b (127.6.212.1:42317), faaacac03e5747f48f17a2b84561a5d8 (127.6.212.2:34887)
I20260924 11:34:36.897027 14446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:36.887616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:56007 (local address 127.6.212.3:45033)
0924 11:34:36.887828 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:36.887836 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:36.889261 (+ 1425us) server_negotiation.cc:407] Connection header received
0924 11:34:36.889439 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:36.889444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:36.889514 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:36.889601 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:34:36.890272 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:36.891378 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:36.894167 (+ 2789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:36.895815 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:36.895942 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:36.896043 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:36.896551 (+ 508us) server_negotiation.cc:299] Negotiation successful
0924 11:34:36.896691 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":62}
I20260924 11:34:36.898133 14442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:36.887554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:35127 (local address 127.6.212.1:42317)
0924 11:34:36.887802 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:36.887812 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:36.888468 (+ 656us) server_negotiation.cc:407] Connection header received
0924 11:34:36.889184 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:36.889189 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:36.889315 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:36.889432 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:34:36.889999 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:36.891115 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:36.895526 (+ 4411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:36.896960 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:36.897104 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:36.897199 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:36.897716 (+ 517us) server_negotiation.cc:299] Negotiation successful
0924 11:34:36.897902 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":104,"server-negotiator.queue_time_us":70}
I20260924 11:34:36.898125 14457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:36.887968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.3:45033 (local address 127.6.212.2:56007)
0924 11:34:36.889134 (+ 1166us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:36.889160 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:36.889297 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:36.889736 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:36.889739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:36.889758 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:34:36.890105 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:36.890116 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:36.891528 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:36.891532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:36.893976 (+ 2444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:36.893990 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:36.896162 (+ 2172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:36.896169 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:36.896362 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:36.896403 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 11:34:36.897609 (+ 1206us) client_negotiation.cc:240] Negotiation successful
0924 11:34:36.897759 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1045,"thread_start_us":373,"threads_started":1}
I20260924 11:34:36.898484 14456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:36.887436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:42317 (local address 127.6.212.2:35127)
0924 11:34:36.888352 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:36.888374 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:36.889029 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:36.889457 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:36.889461 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:36.889483 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:34:36.889821 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:36.889834 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:36.893128 (+ 3294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:36.893133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:36.895332 (+ 2199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:36.895348 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:36.897345 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:36.897351 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:36.897536 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:36.897570 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 11:34:36.897856 (+ 286us) client_negotiation.cc:240] Negotiation successful
0924 11:34:36.898013 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":394,"threads_started":1}
I20260924 11:34:36.899140 14419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:36.890850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.2:34887 (local address 127.6.212.3:48847)
0924 11:34:36.891540 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:36.891558 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:36.891644 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:36.892408 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:36.892412 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:36.892442 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:34:36.892629 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:36.892639 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:36.894124 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:36.894129 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:36.896449 (+ 2320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:36.896464 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:36.898181 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:36.898187 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:36.898405 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:36.898439 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 11:34:36.898557 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 11:34:36.898839 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":117}
I20260924 11:34:36.899231 14368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea281391bc24507a335f41a18079c0c" candidate_uuid: "faaacac03e5747f48f17a2b84561a5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1b300a0baf545cebef745d22579f93e" is_pre_election: true
I20260924 11:34:36.899509 14079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea281391bc24507a335f41a18079c0c" candidate_uuid: "faaacac03e5747f48f17a2b84561a5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92c42ff207f3406aa06f3984a03fc08b" is_pre_election: true
I20260924 11:34:36.899737 14368 raft_consensus.cc:2454] T 5ea281391bc24507a335f41a18079c0c P b1b300a0baf545cebef745d22579f93e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faaacac03e5747f48f17a2b84561a5d8 in term 0.
I20260924 11:34:36.899946 14444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:36.891852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:48847 (local address 127.6.212.2:34887)
0924 11:34:36.892035 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:36.892041 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:36.892060 (+ 19us) server_negotiation.cc:407] Connection header received
0924 11:34:36.892110 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:36.892113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:36.892185 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:36.892268 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:34:36.892786 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:36.893949 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:36.896640 (+ 2691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:36.897864 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:36.897985 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:36.898078 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:36.899529 (+ 1451us) server_negotiation.cc:299] Negotiation successful
0924 11:34:36.899667 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 11:34:36.900179 14079 raft_consensus.cc:2454] T 5ea281391bc24507a335f41a18079c0c P 92c42ff207f3406aa06f3984a03fc08b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faaacac03e5747f48f17a2b84561a5d8 in term 0.
I20260924 11:34:36.900475 14161 leader_election.cc:302] T 5ea281391bc24507a335f41a18079c0c P faaacac03e5747f48f17a2b84561a5d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1b300a0baf545cebef745d22579f93e, faaacac03e5747f48f17a2b84561a5d8; no voters:
I20260924 11:34:36.900678 14227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea281391bc24507a335f41a18079c0c" candidate_uuid: "b1b300a0baf545cebef745d22579f93e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faaacac03e5747f48f17a2b84561a5d8" is_pre_election: true
I20260924 11:34:36.901074 14227 raft_consensus.cc:2454] T 5ea281391bc24507a335f41a18079c0c P faaacac03e5747f48f17a2b84561a5d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1b300a0baf545cebef745d22579f93e in term 0.
I20260924 11:34:36.901275 14421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:36.890508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:42317 (local address 127.6.212.3:39653)
0924 11:34:36.891139 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:36.891161 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:36.891280 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:36.892326 (+ 1046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:36.892331 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:36.892357 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:34:36.892582 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:36.892601 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:36.894256 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:36.894263 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:36.896529 (+ 2266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:36.896541 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:36.900573 (+ 4032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:36.900579 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:36.900786 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:36.900823 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 11:34:36.900891 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 11:34:36.901027 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":247}
I20260924 11:34:36.901232 14453 raft_consensus.cc:2781] T 5ea281391bc24507a335f41a18079c0c P faaacac03e5747f48f17a2b84561a5d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:34:36.901384 14453 raft_consensus.cc:482] T 5ea281391bc24507a335f41a18079c0c P faaacac03e5747f48f17a2b84561a5d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:34:36.901504 14453 raft_consensus.cc:3037] T 5ea281391bc24507a335f41a18079c0c P faaacac03e5747f48f17a2b84561a5d8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:34:36.901654 14304 leader_election.cc:302] T 5ea281391bc24507a335f41a18079c0c P b1b300a0baf545cebef745d22579f93e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1b300a0baf545cebef745d22579f93e, faaacac03e5747f48f17a2b84561a5d8; no voters:
I20260924 11:34:36.902135 14454 raft_consensus.cc:2781] T 5ea281391bc24507a335f41a18079c0c P b1b300a0baf545cebef745d22579f93e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:34:36.902279 14454 raft_consensus.cc:482] T 5ea281391bc24507a335f41a18079c0c P b1b300a0baf545cebef745d22579f93e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:34:36.902354 14458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:36.890696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:39653 (local address 127.6.212.1:42317)
0924 11:34:36.892032 (+ 1336us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:36.892038 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:36.892059 (+ 21us) server_negotiation.cc:407] Connection header received
0924 11:34:36.892108 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:36.892113 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:36.892178 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:36.892726 (+ 548us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:34:36.892937 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:36.894086 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:36.896822 (+ 2736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:36.900325 (+ 3503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:36.901665 (+ 1340us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:36.901833 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:36.901888 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 11:34:36.902038 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":1215,"thread_start_us":1043,"threads_started":1}
I20260924 11:34:36.902402 14454 raft_consensus.cc:3037] T 5ea281391bc24507a335f41a18079c0c P b1b300a0baf545cebef745d22579f93e [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:34:36.902987 14079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea281391bc24507a335f41a18079c0c" candidate_uuid: "b1b300a0baf545cebef745d22579f93e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92c42ff207f3406aa06f3984a03fc08b" is_pre_election: true
I20260924 11:34:36.903208 14079 raft_consensus.cc:2454] T 5ea281391bc24507a335f41a18079c0c P 92c42ff207f3406aa06f3984a03fc08b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1b300a0baf545cebef745d22579f93e in term 0.
I20260924 11:34:36.904603 14453 raft_consensus.cc:504] T 5ea281391bc24507a335f41a18079c0c P faaacac03e5747f48f17a2b84561a5d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c42ff207f3406aa06f3984a03fc08b" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42317 } } peers { permanent_uuid: "faaacac03e5747f48f17a2b84561a5d8" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 34887 } } peers { permanent_uuid: "b1b300a0baf545cebef745d22579f93e" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 45033 } }
I20260924 11:34:36.905612 14079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea281391bc24507a335f41a18079c0c" candidate_uuid: "faaacac03e5747f48f17a2b84561a5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92c42ff207f3406aa06f3984a03fc08b"
I20260924 11:34:36.905799 14079 raft_consensus.cc:3037] T 5ea281391bc24507a335f41a18079c0c P 92c42ff207f3406aa06f3984a03fc08b [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:34:36.906113 14453 leader_election.cc:288] T 5ea281391bc24507a335f41a18079c0c P faaacac03e5747f48f17a2b84561a5d8 [CANDIDATE]: Term 1 election: Requested vote from peers 92c42ff207f3406aa06f3984a03fc08b (127.6.212.1:42317), b1b300a0baf545cebef745d22579f93e (127.6.212.3:45033)
I20260924 11:34:36.906142 14368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea281391bc24507a335f41a18079c0c" candidate_uuid: "faaacac03e5747f48f17a2b84561a5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1b300a0baf545cebef745d22579f93e"
I20260924 11:34:36.905623 14454 raft_consensus.cc:504] T 5ea281391bc24507a335f41a18079c0c P b1b300a0baf545cebef745d22579f93e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c42ff207f3406aa06f3984a03fc08b" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42317 } } peers { permanent_uuid: "faaacac03e5747f48f17a2b84561a5d8" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 34887 } } peers { permanent_uuid: "b1b300a0baf545cebef745d22579f93e" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 45033 } }
I20260924 11:34:36.906716 14368 raft_consensus.cc:2379] T 5ea281391bc24507a335f41a18079c0c P b1b300a0baf545cebef745d22579f93e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate faaacac03e5747f48f17a2b84561a5d8 in current term 1: Already voted for candidate b1b300a0baf545cebef745d22579f93e in this term.
I20260924 11:34:36.907297 14078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea281391bc24507a335f41a18079c0c" candidate_uuid: "b1b300a0baf545cebef745d22579f93e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92c42ff207f3406aa06f3984a03fc08b"
I20260924 11:34:36.907958 14454 leader_election.cc:288] T 5ea281391bc24507a335f41a18079c0c P b1b300a0baf545cebef745d22579f93e [CANDIDATE]: Term 1 election: Requested vote from peers 92c42ff207f3406aa06f3984a03fc08b (127.6.212.1:42317), faaacac03e5747f48f17a2b84561a5d8 (127.6.212.2:34887)
I20260924 11:34:36.908519 14227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea281391bc24507a335f41a18079c0c" candidate_uuid: "b1b300a0baf545cebef745d22579f93e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faaacac03e5747f48f17a2b84561a5d8"
I20260924 11:34:36.908798 14227 raft_consensus.cc:2379] T 5ea281391bc24507a335f41a18079c0c P faaacac03e5747f48f17a2b84561a5d8 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b1b300a0baf545cebef745d22579f93e in current term 1: Already voted for candidate faaacac03e5747f48f17a2b84561a5d8 in this term.
I20260924 11:34:36.909014 14079 raft_consensus.cc:2454] T 5ea281391bc24507a335f41a18079c0c P 92c42ff207f3406aa06f3984a03fc08b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faaacac03e5747f48f17a2b84561a5d8 in term 1.
I20260924 11:34:36.909339 14304 leader_election.cc:302] T 5ea281391bc24507a335f41a18079c0c P b1b300a0baf545cebef745d22579f93e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b1b300a0baf545cebef745d22579f93e; no voters: 92c42ff207f3406aa06f3984a03fc08b, faaacac03e5747f48f17a2b84561a5d8
I20260924 11:34:36.909415 14163 leader_election.cc:302] T 5ea281391bc24507a335f41a18079c0c P faaacac03e5747f48f17a2b84561a5d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 92c42ff207f3406aa06f3984a03fc08b, faaacac03e5747f48f17a2b84561a5d8; no voters: b1b300a0baf545cebef745d22579f93e
I20260924 11:34:36.909745 14453 raft_consensus.cc:2781] T 5ea281391bc24507a335f41a18079c0c P faaacac03e5747f48f17a2b84561a5d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:34:36.909811 14454 raft_consensus.cc:2726] T 5ea281391bc24507a335f41a18079c0c P b1b300a0baf545cebef745d22579f93e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260924 11:34:36.911693 14453 raft_consensus.cc:686] T 5ea281391bc24507a335f41a18079c0c P faaacac03e5747f48f17a2b84561a5d8 [term 1 LEADER]: Becoming Leader. State: Replica: faaacac03e5747f48f17a2b84561a5d8, State: Running, Role: LEADER
I20260924 11:34:36.912297 14453 consensus_queue.cc:237] T 5ea281391bc24507a335f41a18079c0c P faaacac03e5747f48f17a2b84561a5d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c42ff207f3406aa06f3984a03fc08b" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42317 } } peers { permanent_uuid: "faaacac03e5747f48f17a2b84561a5d8" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 34887 } } peers { permanent_uuid: "b1b300a0baf545cebef745d22579f93e" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 45033 } }
I20260924 11:34:36.918123 13159 catalog_manager.cc:5495] T 5ea281391bc24507a335f41a18079c0c P faaacac03e5747f48f17a2b84561a5d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to faaacac03e5747f48f17a2b84561a5d8 (127.6.212.2). New cstate: current_term: 1 leader_uuid: "faaacac03e5747f48f17a2b84561a5d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c42ff207f3406aa06f3984a03fc08b" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "faaacac03e5747f48f17a2b84561a5d8" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 34887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1b300a0baf545cebef745d22579f93e" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 45033 } health_report { overall_health: UNKNOWN } } }
W20260924 11:34:36.931922 14275 tablet.cc:2369] T 5ea281391bc24507a335f41a18079c0c P faaacac03e5747f48f17a2b84561a5d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:34:36.963119 14444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:36.957680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42106 (local address 127.6.212.2:34887)
0924 11:34:36.957858 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:36.957863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:36.957882 (+ 19us) server_negotiation.cc:407] Connection header received
0924 11:34:36.958144 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:36.958149 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:36.958230 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:36.958363 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:34:36.958716 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:36.959564 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:36.960934 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:36.961422 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:36.961517 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:36.962080 (+ 563us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:34:36.962592 (+ 512us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:34:36.962700 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:36.962792 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 11:34:36.962857 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 11:34:36.971349 14079 raft_consensus.cc:1261] T 5ea281391bc24507a335f41a18079c0c P 92c42ff207f3406aa06f3984a03fc08b [term 1 FOLLOWER]: Refusing update from remote peer faaacac03e5747f48f17a2b84561a5d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:34:36.971612 14368 raft_consensus.cc:1261] T 5ea281391bc24507a335f41a18079c0c P b1b300a0baf545cebef745d22579f93e [term 1 FOLLOWER]: Refusing update from remote peer faaacac03e5747f48f17a2b84561a5d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:34:36.972374 14459 consensus_queue.cc:1035] T 5ea281391bc24507a335f41a18079c0c P faaacac03e5747f48f17a2b84561a5d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92c42ff207f3406aa06f3984a03fc08b" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:34:36.972782 14453 consensus_queue.cc:1035] T 5ea281391bc24507a335f41a18079c0c P faaacac03e5747f48f17a2b84561a5d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1b300a0baf545cebef745d22579f93e" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 45033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:34:37.006139 13159 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:34:37.006444 13159 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:34:37.013176 13159 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f32c56b9b2b94674b1fde17e4e460006: Sending DeleteTablet for 3 replicas of tablet 5ea281391bc24507a335f41a18079c0c
I20260924 11:34:37.014534 14348 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ea281391bc24507a335f41a18079c0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:34:37 UTC) from {username='oryx', principal='oryx/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:41866
I20260924 11:34:37.014941 14059 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ea281391bc24507a335f41a18079c0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:34:37 UTC) from {username='oryx', principal='oryx/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:46044
I20260924 11:34:37.015004 14207 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ea281391bc24507a335f41a18079c0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:34:37 UTC) from {username='oryx', principal='oryx/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:42092
I20260924 11:34:37.019169 14473 tablet_replica.cc:330] T 5ea281391bc24507a335f41a18079c0c P faaacac03e5747f48f17a2b84561a5d8: stopping tablet replica
I20260924 11:34:37.019719 14473 raft_consensus.cc:2229] T 5ea281391bc24507a335f41a18079c0c P faaacac03e5747f48f17a2b84561a5d8 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:34:37.020004 14474 tablet_replica.cc:330] T 5ea281391bc24507a335f41a18079c0c P 92c42ff207f3406aa06f3984a03fc08b: stopping tablet replica
I20260924 11:34:37.020437 14473 raft_consensus.cc:2258] T 5ea281391bc24507a335f41a18079c0c P faaacac03e5747f48f17a2b84561a5d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:34:37.020484 14474 raft_consensus.cc:2229] T 5ea281391bc24507a335f41a18079c0c P 92c42ff207f3406aa06f3984a03fc08b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:34:37.021104 14474 raft_consensus.cc:2258] T 5ea281391bc24507a335f41a18079c0c P 92c42ff207f3406aa06f3984a03fc08b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:34:37.022684 14473 ts_tablet_manager.cc:1905] T 5ea281391bc24507a335f41a18079c0c P faaacac03e5747f48f17a2b84561a5d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:34:37.023089 14474 ts_tablet_manager.cc:1905] T 5ea281391bc24507a335f41a18079c0c P 92c42ff207f3406aa06f3984a03fc08b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:34:37.030567 14474 ts_tablet_manager.cc:1918] T 5ea281391bc24507a335f41a18079c0c P 92c42ff207f3406aa06f3984a03fc08b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:34:37.030571 14473 ts_tablet_manager.cc:1918] T 5ea281391bc24507a335f41a18079c0c P faaacac03e5747f48f17a2b84561a5d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:34:37.030755 14473 log.cc:1196] T 5ea281391bc24507a335f41a18079c0c P faaacac03e5747f48f17a2b84561a5d8: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-1/wal/wals/5ea281391bc24507a335f41a18079c0c
I20260924 11:34:37.030944 14474 log.cc:1196] T 5ea281391bc24507a335f41a18079c0c P 92c42ff207f3406aa06f3984a03fc08b: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-0/wal/wals/5ea281391bc24507a335f41a18079c0c
I20260924 11:34:37.031428 14473 ts_tablet_manager.cc:1939] T 5ea281391bc24507a335f41a18079c0c P faaacac03e5747f48f17a2b84561a5d8: Deleting consensus metadata
I20260924 11:34:37.031713 14474 ts_tablet_manager.cc:1939] T 5ea281391bc24507a335f41a18079c0c P 92c42ff207f3406aa06f3984a03fc08b: Deleting consensus metadata
I20260924 11:34:37.033420 13146 catalog_manager.cc:4841] TS 92c42ff207f3406aa06f3984a03fc08b (127.6.212.1:42317): tablet 5ea281391bc24507a335f41a18079c0c (table test-table [id=1cb515a8b01b4250af3c53f00fc863e8]) successfully deleted
I20260924 11:34:37.033852 13146 catalog_manager.cc:4841] TS faaacac03e5747f48f17a2b84561a5d8 (127.6.212.2:34887): tablet 5ea281391bc24507a335f41a18079c0c (table test-table [id=1cb515a8b01b4250af3c53f00fc863e8]) successfully deleted
I20260924 11:34:37.036523 14472 tablet_replica.cc:330] T 5ea281391bc24507a335f41a18079c0c P b1b300a0baf545cebef745d22579f93e: stopping tablet replica
I20260924 11:34:37.036742 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 13127
I20260924 11:34:37.037048 14472 raft_consensus.cc:2229] T 5ea281391bc24507a335f41a18079c0c P b1b300a0baf545cebef745d22579f93e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:34:37.037575 14472 raft_consensus.cc:2258] T 5ea281391bc24507a335f41a18079c0c P b1b300a0baf545cebef745d22579f93e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:34:37.038479 13189 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:34:37.039803 14472 ts_tablet_manager.cc:1905] T 5ea281391bc24507a335f41a18079c0c P b1b300a0baf545cebef745d22579f93e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:34:37.047257 14472 ts_tablet_manager.cc:1918] T 5ea281391bc24507a335f41a18079c0c P b1b300a0baf545cebef745d22579f93e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:34:37.047473 14472 log.cc:1196] T 5ea281391bc24507a335f41a18079c0c P b1b300a0baf545cebef745d22579f93e: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-2/wal/wals/5ea281391bc24507a335f41a18079c0c
I20260924 11:34:37.048141 14472 ts_tablet_manager.cc:1939] T 5ea281391bc24507a335f41a18079c0c P b1b300a0baf545cebef745d22579f93e: Deleting consensus metadata
I20260924 11:34:37.053838 14427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:37.018442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43666 (local address 127.6.212.61:38525)
0924 11:34:37.018713 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:37.018723 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:37.035246 (+ 16523us) server_negotiation.cc:407] Connection header received
0924 11:34:37.035413 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:37.035422 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:37.035532 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:37.035642 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:37.037890 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:37.038994 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:37.041495 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:37.042219 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:37.042366 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:37.046747 (+ 4381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:37.046754 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:37.046758 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:37.046802 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:37.050953 (+ 4151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:37.051766 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:37.051771 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:37.051777 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:37.051971 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:37.052354 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:37.052366 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:37.052369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:37.052648 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:37.052910 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:37.053276 (+ 366us) server_negotiation.cc:299] Negotiation successful
0924 11:34:37.053499 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 11:34:37.067847 14420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:37.035106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52918 (local address 127.6.212.60:43695)
0924 11:34:37.035307 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:37.035315 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:37.054011 (+ 18696us) server_negotiation.cc:407] Connection header received
0924 11:34:37.054199 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:37.054205 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:37.054295 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:37.054370 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:37.056786 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:37.057518 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:37.058807 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:37.059246 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:37.059421 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:37.062082 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:37.062091 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:37.062100 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:37.062133 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:37.065060 (+ 2927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:37.065730 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:37.065736 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:37.065741 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:37.065865 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:37.066251 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:37.066256 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:37.066259 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:37.066543 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:37.066862 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:37.067209 (+ 347us) server_negotiation.cc:299] Negotiation successful
0924 11:34:37.067464 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 11:34:37.197584 13144 catalog_manager.cc:4841] TS b1b300a0baf545cebef745d22579f93e (127.6.212.3:45033): tablet 5ea281391bc24507a335f41a18079c0c (table test-table [id=1cb515a8b01b4250af3c53f00fc863e8]) successfully deleted
I20260924 11:34:37.198623 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 13127
I20260924 11:34:37.262550 14420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:37.235066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52930 (local address 127.6.212.60:43695)
0924 11:34:37.235324 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:37.235332 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:37.242979 (+ 7647us) server_negotiation.cc:407] Connection header received
0924 11:34:37.243231 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:37.243238 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:37.243386 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:37.243507 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:37.246068 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:37.247269 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:37.249818 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:37.250668 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:37.250781 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:37.255326 (+ 4545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:37.255334 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:37.255340 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:37.255429 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:37.258688 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:37.259597 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:37.259604 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:37.259608 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:37.259773 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:37.260312 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:37.260320 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:37.260323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:37.260741 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:37.261007 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:37.261701 (+ 694us) server_negotiation.cc:299] Negotiation successful
0924 11:34:37.262121 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 11:34:37.266062 14427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:37.241356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43676 (local address 127.6.212.61:38525)
0924 11:34:37.241636 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:37.241645 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:37.249176 (+ 7531us) server_negotiation.cc:407] Connection header received
0924 11:34:37.249502 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:37.249510 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:37.249619 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:37.249694 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:37.252575 (+ 2881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:37.253554 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:37.255146 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:37.255921 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:37.256082 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:37.259280 (+ 3198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:37.259288 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:37.259294 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:37.259355 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:37.262819 (+ 3464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:37.263755 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:37.263761 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:37.263765 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:37.263935 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:37.264408 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:37.264415 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:37.264419 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:37.264801 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:37.265066 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:37.265429 (+ 363us) server_negotiation.cc:299] Negotiation successful
0924 11:34:37.265671 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260924 11:34:37.992200 14484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:37.991324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.6.212.2:38031)
0924 11:34:37.991883 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:37.992036 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":395,"threads_started":1}
W20260924 11:34:37.992609 14273 heartbeater.cc:644] Failed to heartbeat to 127.6.212.62:42625 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
I20260924 11:34:37.994742 14485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:37.993913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.6.212.3:49199)
0924 11:34:37.994381 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:37.994522 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":250,"threads_started":1}
W20260924 11:34:37.995133 14414 heartbeater.cc:644] Failed to heartbeat to 127.6.212.62:42625 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
I20260924 11:34:37.997752 14486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:37.997011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.6.212.1:40149)
0924 11:34:37.997420 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:37.997539 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":264,"threads_started":1}
W20260924 11:34:37.998356 14125 heartbeater.cc:644] Failed to heartbeat to 127.6.212.62:42625 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
I20260924 11:34:38.514158 14488 raft_consensus.cc:482] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f32c56b9b2b94674b1fde17e4e460006)
I20260924 11:34:38.514539 14488 raft_consensus.cc:504] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } peers { permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 38525 } attrs { promote: false } } peers { permanent_uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 43695 } attrs { promote: false } }
I20260924 11:34:38.514782 14487 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f32c56b9b2b94674b1fde17e4e460006)
I20260924 11:34:38.515161 14487 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } peers { permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 38525 } attrs { promote: false } } peers { permanent_uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 43695 } attrs { promote: false } }
I20260924 11:34:38.516218 14488 leader_election.cc:288] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f32c56b9b2b94674b1fde17e4e460006 (127.6.212.62:42625), bbe9da2f8b0e4ed79c32eb5d554a06e5 (127.6.212.60:43695)
I20260924 11:34:38.522743 14487 leader_election.cc:288] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f32c56b9b2b94674b1fde17e4e460006 (127.6.212.62:42625), d211d01c60f2417bb346495d00f5593f (127.6.212.61:38525)
I20260924 11:34:38.524991 14490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:38.516405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.0.0.1:50512)
0924 11:34:38.524448 (+ 8043us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:38.524655 (+ 207us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7864,"threads_started":1}
I20260924 11:34:38.529330 14492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:38.516956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.0.0.1:50514)
0924 11:34:38.528862 (+ 11906us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:38.529060 (+ 198us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11768,"threads_started":1}
W20260924 11:34:38.529701 13742 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
W20260924 11:34:38.529824 13928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
I20260924 11:34:38.537134 14490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:38.516474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:43695 (local address 127.0.0.1:52944)
0924 11:34:38.526370 (+ 9896us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:38.526394 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:38.526564 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:38.530339 (+ 3775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:38.530343 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:38.530367 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:34:38.530718 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:38.530734 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:38.532403 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:38.532411 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:38.534487 (+ 2076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:38.534513 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:38.536334 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:38.536340 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:38.536545 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:38.536575 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 11:34:38.536658 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 11:34:38.536794 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9776,"mutex_wait_us":1,"thread_start_us":7663,"threads_started":1}
I20260924 11:34:38.537218 14491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:38.516780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:38525 (local address 127.0.0.1:43678)
0924 11:34:38.528153 (+ 11373us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:38.528175 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:38.528340 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:38.529960 (+ 1620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:38.529964 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:38.529987 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:34:38.530350 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:38.530372 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:38.532215 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:38.532220 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:38.534510 (+ 2290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:38.534522 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:38.536331 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:38.536337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:38.536526 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:38.536555 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 11:34:38.536650 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 11:34:38.536788 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11196,"mutex_wait_us":4,"thread_start_us":11015,"threads_started":1}
I20260924 11:34:38.537712 14492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:38.537321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.0.0.1:50528)
0924 11:34:38.537493 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:38.537570 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 11:34:38.538058 14494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:38.528135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52944 (local address 127.6.212.60:43695)
0924 11:34:38.529772 (+ 1637us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:38.529779 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:38.529796 (+ 17us) server_negotiation.cc:407] Connection header received
0924 11:34:38.529885 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:38.529890 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:38.529959 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:38.530058 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:34:38.530940 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:38.532247 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:38.534687 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:38.536026 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:38.536128 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:38.536223 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:38.537422 (+ 1199us) server_negotiation.cc:299] Negotiation successful
0924 11:34:38.537553 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1516,"spinlock_wait_cycles":2304,"thread_start_us":1711,"threads_started":1}
I20260924 11:34:38.538148 14493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:38.524442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43678 (local address 127.6.212.61:38525)
0924 11:34:38.529375 (+ 4933us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:38.529383 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:38.529408 (+ 25us) server_negotiation.cc:407] Connection header received
0924 11:34:38.529513 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:38.529520 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:38.529600 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:38.529712 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:34:38.530643 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:38.532032 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:38.534668 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:38.536007 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:38.536129 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:38.536225 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:38.537680 (+ 1455us) server_negotiation.cc:299] Negotiation successful
0924 11:34:38.537856 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4772,"thread_start_us":4584,"threads_started":1}
W20260924 11:34:38.538115 13928 leader_election.cc:334] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f32c56b9b2b94674b1fde17e4e460006 (127.6.212.62:42625): Network error: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
I20260924 11:34:38.538573 13953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d211d01c60f2417bb346495d00f5593f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5" is_pre_election: true
I20260924 11:34:38.538659 13766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d211d01c60f2417bb346495d00f5593f" is_pre_election: true
I20260924 11:34:38.538964 13953 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d211d01c60f2417bb346495d00f5593f in term 1.
I20260924 11:34:38.538959 13766 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbe9da2f8b0e4ed79c32eb5d554a06e5 in term 1.
I20260924 11:34:38.539520 13743 leader_election.cc:302] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbe9da2f8b0e4ed79c32eb5d554a06e5, d211d01c60f2417bb346495d00f5593f; no voters:
I20260924 11:34:38.539542 13930 leader_election.cc:302] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bbe9da2f8b0e4ed79c32eb5d554a06e5, d211d01c60f2417bb346495d00f5593f; no voters: f32c56b9b2b94674b1fde17e4e460006
I20260924 11:34:38.539891 14487 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 11:34:38.539911 14488 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 11:34:38.540011 14487 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f32c56b9b2b94674b1fde17e4e460006)
I20260924 11:34:38.540024 14488 raft_consensus.cc:482] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [term 1 FOLLOWER]: Starting leader election (detected failure of leader f32c56b9b2b94674b1fde17e4e460006)
I20260924 11:34:38.540112 14488 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [term 1 FOLLOWER]: Advancing to term 2
I20260924 11:34:38.540102 14487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5 [term 1 FOLLOWER]: Advancing to term 2
I20260924 11:34:38.540489 14490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:38.540027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.0.0.1:50548)
0924 11:34:38.540233 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:38.540326 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260924 11:34:38.540688 13742 leader_election.cc:334] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f32c56b9b2b94674b1fde17e4e460006 (127.6.212.62:42625): Network error: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
I20260924 11:34:38.542487 14487 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } peers { permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 38525 } attrs { promote: false } } peers { permanent_uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 43695 } attrs { promote: false } }
I20260924 11:34:38.543088 14487 leader_election.cc:288] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5 [CANDIDATE]: Term 2 election: Requested vote from peers f32c56b9b2b94674b1fde17e4e460006 (127.6.212.62:42625), d211d01c60f2417bb346495d00f5593f (127.6.212.61:38525)
I20260924 11:34:38.543172 14488 raft_consensus.cc:504] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } peers { permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 38525 } attrs { promote: false } } peers { permanent_uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 43695 } attrs { promote: false } }
I20260924 11:34:38.543587 13766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d211d01c60f2417bb346495d00f5593f"
I20260924 11:34:38.543910 13766 raft_consensus.cc:2379] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate bbe9da2f8b0e4ed79c32eb5d554a06e5 in current term 2: Already voted for candidate d211d01c60f2417bb346495d00f5593f in this term.
I20260924 11:34:38.544296 14491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:38.543406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.0.0.1:50558)
0924 11:34:38.544040 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:38.544139 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443}
I20260924 11:34:38.544484 14488 leader_election.cc:288] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [CANDIDATE]: Term 2 election: Requested vote from peers f32c56b9b2b94674b1fde17e4e460006 (127.6.212.62:42625), bbe9da2f8b0e4ed79c32eb5d554a06e5 (127.6.212.60:43695)
I20260924 11:34:38.545084 13953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d211d01c60f2417bb346495d00f5593f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5"
I20260924 11:34:38.545375 13953 raft_consensus.cc:2379] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d211d01c60f2417bb346495d00f5593f in current term 2: Already voted for candidate bbe9da2f8b0e4ed79c32eb5d554a06e5 in this term.
I20260924 11:34:38.545552 14491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:38.545087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.0.0.1:50570)
0924 11:34:38.545299 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:38.545410 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260924 11:34:38.545570 14490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:38.545148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.0.0.1:50576)
0924 11:34:38.545366 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:38.545456 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
W20260924 11:34:38.545730 13928 leader_election.cc:334] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f32c56b9b2b94674b1fde17e4e460006 (127.6.212.62:42625): Network error: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
I20260924 11:34:38.545845 13928 leader_election.cc:302] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbe9da2f8b0e4ed79c32eb5d554a06e5; no voters: d211d01c60f2417bb346495d00f5593f, f32c56b9b2b94674b1fde17e4e460006
I20260924 11:34:38.546156 14487 raft_consensus.cc:2726] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 11:34:38.546571 14490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:38.546192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.0.0.1:50590)
0924 11:34:38.546382 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:38.546444 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260924 11:34:38.546798 13742 leader_election.cc:334] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f32c56b9b2b94674b1fde17e4e460006 (127.6.212.62:42625): Network error: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
I20260924 11:34:38.546917 13742 leader_election.cc:302] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d211d01c60f2417bb346495d00f5593f; no voters: bbe9da2f8b0e4ed79c32eb5d554a06e5, f32c56b9b2b94674b1fde17e4e460006
I20260924 11:34:38.547161 14488 raft_consensus.cc:2726] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 11:34:38.994124 14497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:38.993332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.6.212.2:59723)
0924 11:34:38.993881 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:38.993960 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":346,"threads_started":1}
I20260924 11:34:38.996619 14498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:38.995847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.6.212.3:50791)
0924 11:34:38.996404 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:38.996476 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":296,"threads_started":1}
I20260924 11:34:38.999858 14499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:38.999079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.6.212.1:34595)
0924 11:34:38.999612 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:38.999700 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":329,"threads_started":1}
I20260924 11:34:39.995749 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:39.994905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.6.212.2:32959)
0924 11:34:39.995495 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:39.995582 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":391,"threads_started":1}
I20260924 11:34:39.998132 14501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:39.997453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.6.212.3:41043)
0924 11:34:39.997868 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:39.997980 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":205,"threads_started":1}
I20260924 11:34:40.001223 14502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:40.000642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.6.212.1:34325)
0924 11:34:40.001005 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:40.001083 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":187,"threads_started":1}
I20260924 11:34:40.109717 14503 raft_consensus.cc:482] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:34:40.109987 14503 raft_consensus.cc:504] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } peers { permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 38525 } attrs { promote: false } } peers { permanent_uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 43695 } attrs { promote: false } }
I20260924 11:34:40.110598 14503 leader_election.cc:288] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f32c56b9b2b94674b1fde17e4e460006 (127.6.212.62:42625), bbe9da2f8b0e4ed79c32eb5d554a06e5 (127.6.212.60:43695)
I20260924 11:34:40.111240 13953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d211d01c60f2417bb346495d00f5593f" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5" is_pre_election: true
I20260924 11:34:40.111600 13953 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d211d01c60f2417bb346495d00f5593f in term 2.
I20260924 11:34:40.111734 14504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:40.110907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.0.0.1:50798)
0924 11:34:40.111340 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:40.111494 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":239,"threads_started":1}
I20260924 11:34:40.112099 13743 leader_election.cc:302] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbe9da2f8b0e4ed79c32eb5d554a06e5, d211d01c60f2417bb346495d00f5593f; no voters:
I20260924 11:34:40.112449 14503 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 11:34:40.112614 14503 raft_consensus.cc:482] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:34:40.112747 14503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [term 2 FOLLOWER]: Advancing to term 3
I20260924 11:34:40.113305 14504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:40.112904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.0.0.1:50806)
0924 11:34:40.113076 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:40.113159 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 11:34:40.113502 13742 leader_election.cc:334] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f32c56b9b2b94674b1fde17e4e460006 (127.6.212.62:42625): Network error: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
I20260924 11:34:40.115176 14503 raft_consensus.cc:504] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } peers { permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 38525 } attrs { promote: false } } peers { permanent_uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 43695 } attrs { promote: false } }
I20260924 11:34:40.115705 14503 leader_election.cc:288] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [CANDIDATE]: Term 3 election: Requested vote from peers f32c56b9b2b94674b1fde17e4e460006 (127.6.212.62:42625), bbe9da2f8b0e4ed79c32eb5d554a06e5 (127.6.212.60:43695)
I20260924 11:34:40.116297 14504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:40.115876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.0.0.1:50812)
0924 11:34:40.116053 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:40.116142 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 11:34:40.116448 13953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d211d01c60f2417bb346495d00f5593f" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5"
I20260924 11:34:40.116680 13953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5 [term 2 FOLLOWER]: Advancing to term 3
I20260924 11:34:40.117363 14504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:40.116883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.0.0.1:50822)
0924 11:34:40.117103 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:40.117189 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260924 11:34:40.117528 13742 leader_election.cc:334] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer f32c56b9b2b94674b1fde17e4e460006 (127.6.212.62:42625): Network error: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
I20260924 11:34:40.119330 13953 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d211d01c60f2417bb346495d00f5593f in term 3.
I20260924 11:34:40.119827 13743 leader_election.cc:302] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bbe9da2f8b0e4ed79c32eb5d554a06e5, d211d01c60f2417bb346495d00f5593f; no voters: f32c56b9b2b94674b1fde17e4e460006
I20260924 11:34:40.120090 14503 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [term 3 FOLLOWER]: Leader election won for term 3
I20260924 11:34:40.120505 14503 raft_consensus.cc:686] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [term 3 LEADER]: Becoming Leader. State: Replica: d211d01c60f2417bb346495d00f5593f, State: Running, Role: LEADER
I20260924 11:34:40.120877 14503 consensus_queue.cc:237] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } peers { permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 38525 } attrs { promote: false } } peers { permanent_uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 43695 } attrs { promote: false } }
I20260924 11:34:40.122551 14506 sys_catalog.cc:455] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d211d01c60f2417bb346495d00f5593f. Latest consensus state: current_term: 3 leader_uuid: "d211d01c60f2417bb346495d00f5593f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } peers { permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 38525 } attrs { promote: false } } peers { permanent_uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 43695 } attrs { promote: false } } }
I20260924 11:34:40.122915 14506 sys_catalog.cc:458] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [sys.catalog]: This master's current role is: LEADER
I20260924 11:34:40.123587 14508 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:34:40.126950 14508 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:34:40.127802 14508 catalog_manager.cc:1259] Loaded cluster ID: 667ded6d01e14754a126c1667f74f666
I20260924 11:34:40.127875 14508 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:34:40.128997 14508 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:34:40.129828 14508 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f: Loaded TSK: 0
I20260924 11:34:40.130679 14508 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:34:40.142290 13756 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43676:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:34:40.149402 13756 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:34:40.153618 14504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:40.153146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.0.0.1:50840)
0924 11:34:40.153364 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:40.153454 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260924 11:34:40.153622 13953 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5 [term 3 FOLLOWER]: Refusing update from remote peer d211d01c60f2417bb346495d00f5593f: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260924 11:34:40.154320 14506 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 43695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260924 11:34:40.154649 14504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:40.154269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.0.0.1:50846)
0924 11:34:40.154433 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:40.154509 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 11:34:40.154835 13742 consensus_peers.cc:487] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f -> Peer f32c56b9b2b94674b1fde17e4e460006 (127.6.212.62:42625): Couldn't send request to peer f32c56b9b2b94674b1fde17e4e460006. Status: Network error: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 11:34:40.156719 14510 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d211d01c60f2417bb346495d00f5593f. Latest consensus state: current_term: 3 leader_uuid: "d211d01c60f2417bb346495d00f5593f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } peers { permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 38525 } attrs { promote: false } } peers { permanent_uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 43695 } attrs { promote: false } } }
I20260924 11:34:40.156944 14510 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:34:40.159025 14503 sys_catalog.cc:455] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "d211d01c60f2417bb346495d00f5593f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } peers { permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 38525 } attrs { promote: false } } peers { permanent_uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 43695 } attrs { promote: false } } }
I20260924 11:34:40.159318 14503 sys_catalog.cc:458] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f [sys.catalog]: This master's current role is: LEADER
I20260924 11:34:40.159473 14510 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "d211d01c60f2417bb346495d00f5593f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "f32c56b9b2b94674b1fde17e4e460006" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42625 } } peers { permanent_uuid: "d211d01c60f2417bb346495d00f5593f" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 38525 } attrs { promote: false } } peers { permanent_uuid: "bbe9da2f8b0e4ed79c32eb5d554a06e5" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 43695 } attrs { promote: false } } }
I20260924 11:34:40.159729 14510 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbe9da2f8b0e4ed79c32eb5d554a06e5 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:34:40.178478 14504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:40.168322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:42317 (local address 127.0.0.1:46058)
0924 11:34:40.168607 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:40.168626 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:40.168757 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:40.170463 (+ 1706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:40.170469 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:40.170501 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:34:40.170869 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:40.170883 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:40.173066 (+ 2183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:40.173075 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:40.175841 (+ 2766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:40.175861 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:40.177705 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:40.177713 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:40.177900 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:40.177933 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 11:34:40.178033 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 11:34:40.178175 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186}
I20260924 11:34:40.178687 14516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:40.168537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46058 (local address 127.6.212.1:42317)
0924 11:34:40.169996 (+ 1459us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:40.170008 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:40.170038 (+ 30us) server_negotiation.cc:407] Connection header received
0924 11:34:40.170141 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:40.170151 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:40.170266 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:40.170424 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:34:40.171090 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:40.172898 (+ 1808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:40.176041 (+ 3143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:40.177503 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:40.177669 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:40.177804 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:40.178082 (+ 278us) server_negotiation.cc:299] Negotiation successful
0924 11:34:40.178254 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1264,"thread_start_us":1289,"threads_started":1}
I20260924 11:34:40.179665 14059 tablet_service.cc:1467] Processing CreateTablet for tablet 4446e01173504e919b6fabfa38315ab2 (DEFAULT_TABLE table=test-table [id=dd04fe0f4bf34d73b602a5412b4079f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:34:40.180207 14059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4446e01173504e919b6fabfa38315ab2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:34:40.184437 14517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:40.168883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.2:34887 (local address 127.0.0.1:42116)
0924 11:34:40.169557 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:40.169579 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:40.169680 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:40.176445 (+ 6765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:40.176451 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:40.176482 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:34:40.176924 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:40.176938 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:40.178553 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:40.178559 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:40.180853 (+ 2294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:40.180870 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:40.183575 (+ 2705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:40.183581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:40.183818 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:40.183862 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 11:34:40.183923 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 11:34:40.184057 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"mutex_wait_us":5,"thread_start_us":636,"threads_started":1}
I20260924 11:34:40.185353 14518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:40.169353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42116 (local address 127.6.212.2:34887)
0924 11:34:40.175870 (+ 6517us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:40.175876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:40.175895 (+ 19us) server_negotiation.cc:407] Connection header received
0924 11:34:40.175999 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:40.176006 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:40.176098 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:40.176215 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:34:40.177190 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:40.178343 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:40.181044 (+ 2701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:40.183294 (+ 2250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:40.184762 (+ 1468us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:40.184881 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:40.184929 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 11:34:40.185051 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6390,"thread_start_us":6265,"threads_started":1}
I20260924 11:34:40.186180 14207 tablet_service.cc:1467] Processing CreateTablet for tablet 4446e01173504e919b6fabfa38315ab2 (DEFAULT_TABLE table=test-table [id=dd04fe0f4bf34d73b602a5412b4079f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:34:40.186627 14207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4446e01173504e919b6fabfa38315ab2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:34:40.187698 14521 tablet_bootstrap.cc:492] T 4446e01173504e919b6fabfa38315ab2 P 92c42ff207f3406aa06f3984a03fc08b: Bootstrap starting.
I20260924 11:34:40.189484 14521 tablet_bootstrap.cc:654] T 4446e01173504e919b6fabfa38315ab2 P 92c42ff207f3406aa06f3984a03fc08b: Neither blocks nor log segments found. Creating new log.
I20260924 11:34:40.190363 14520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:40.170220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41880 (local address 127.6.212.3:45033)
0924 11:34:40.183041 (+ 12821us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:40.183049 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:40.183073 (+ 24us) server_negotiation.cc:407] Connection header received
0924 11:34:40.183150 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:40.183157 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:40.183239 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:40.183397 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:34:40.184035 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:40.185058 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:40.188038 (+ 2980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:40.189041 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:40.189143 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:40.189237 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:40.189842 (+ 605us) server_negotiation.cc:299] Negotiation successful
0924 11:34:40.190004 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12667,"thread_start_us":12523,"threads_started":1}
I20260924 11:34:40.190357 14519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:40.170094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.3:45033 (local address 127.0.0.1:41880)
0924 11:34:40.177699 (+ 7605us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:40.177716 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:40.177814 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:40.183426 (+ 5612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:40.183433 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:40.183465 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:34:40.183818 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:40.183828 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:40.185334 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:40.185341 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:40.187833 (+ 2492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:40.187850 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:40.189421 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:40.189426 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:40.189618 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:40.189661 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 11:34:40.189786 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 11:34:40.189945 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7453,"thread_start_us":7785,"threads_started":1}
I20260924 11:34:40.191846 14348 tablet_service.cc:1467] Processing CreateTablet for tablet 4446e01173504e919b6fabfa38315ab2 (DEFAULT_TABLE table=test-table [id=dd04fe0f4bf34d73b602a5412b4079f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:34:40.192363 14348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4446e01173504e919b6fabfa38315ab2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:34:40.194902 14523 tablet_bootstrap.cc:492] T 4446e01173504e919b6fabfa38315ab2 P faaacac03e5747f48f17a2b84561a5d8: Bootstrap starting.
I20260924 11:34:40.196328 14521 tablet_bootstrap.cc:492] T 4446e01173504e919b6fabfa38315ab2 P 92c42ff207f3406aa06f3984a03fc08b: No bootstrap required, opened a new log
I20260924 11:34:40.196521 14521 ts_tablet_manager.cc:1397] T 4446e01173504e919b6fabfa38315ab2 P 92c42ff207f3406aa06f3984a03fc08b: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20260924 11:34:40.197136 14523 tablet_bootstrap.cc:654] T 4446e01173504e919b6fabfa38315ab2 P faaacac03e5747f48f17a2b84561a5d8: Neither blocks nor log segments found. Creating new log.
I20260924 11:34:40.197325 14521 raft_consensus.cc:348] T 4446e01173504e919b6fabfa38315ab2 P 92c42ff207f3406aa06f3984a03fc08b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c42ff207f3406aa06f3984a03fc08b" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42317 } } peers { permanent_uuid: "faaacac03e5747f48f17a2b84561a5d8" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 34887 } } peers { permanent_uuid: "b1b300a0baf545cebef745d22579f93e" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 45033 } }
I20260924 11:34:40.197531 14521 raft_consensus.cc:374] T 4446e01173504e919b6fabfa38315ab2 P 92c42ff207f3406aa06f3984a03fc08b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:34:40.197630 14521 raft_consensus.cc:729] T 4446e01173504e919b6fabfa38315ab2 P 92c42ff207f3406aa06f3984a03fc08b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92c42ff207f3406aa06f3984a03fc08b, State: Initialized, Role: FOLLOWER
I20260924 11:34:40.197886 14521 consensus_queue.cc:260] T 4446e01173504e919b6fabfa38315ab2 P 92c42ff207f3406aa06f3984a03fc08b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c42ff207f3406aa06f3984a03fc08b" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42317 } } peers { permanent_uuid: "faaacac03e5747f48f17a2b84561a5d8" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 34887 } } peers { permanent_uuid: "b1b300a0baf545cebef745d22579f93e" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 45033 } }
I20260924 11:34:40.198805 14521 ts_tablet_manager.cc:1428] T 4446e01173504e919b6fabfa38315ab2 P 92c42ff207f3406aa06f3984a03fc08b: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 11:34:40.199652 14502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:40.199073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.6.212.1:60947)
0924 11:34:40.199333 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:40.199455 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260924 11:34:40.200838 14526 tablet_bootstrap.cc:492] T 4446e01173504e919b6fabfa38315ab2 P b1b300a0baf545cebef745d22579f93e: Bootstrap starting.
I20260924 11:34:40.202386 14502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:40.201766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.6.212.1:41897)
0924 11:34:40.202078 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:40.202177 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
I20260924 11:34:40.202397 14126 heartbeater.cc:503] Master 127.6.212.61:38525 was elected leader, sending a full tablet report...
I20260924 11:34:40.203070 14526 tablet_bootstrap.cc:654] T 4446e01173504e919b6fabfa38315ab2 P b1b300a0baf545cebef745d22579f93e: Neither blocks nor log segments found. Creating new log.
I20260924 11:34:40.203222 14523 tablet_bootstrap.cc:492] T 4446e01173504e919b6fabfa38315ab2 P faaacac03e5747f48f17a2b84561a5d8: No bootstrap required, opened a new log
I20260924 11:34:40.203392 14523 ts_tablet_manager.cc:1397] T 4446e01173504e919b6fabfa38315ab2 P faaacac03e5747f48f17a2b84561a5d8: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260924 11:34:40.204229 14523 raft_consensus.cc:348] T 4446e01173504e919b6fabfa38315ab2 P faaacac03e5747f48f17a2b84561a5d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c42ff207f3406aa06f3984a03fc08b" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42317 } } peers { permanent_uuid: "faaacac03e5747f48f17a2b84561a5d8" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 34887 } } peers { permanent_uuid: "b1b300a0baf545cebef745d22579f93e" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 45033 } }
I20260924 11:34:40.204504 14523 raft_consensus.cc:374] T 4446e01173504e919b6fabfa38315ab2 P faaacac03e5747f48f17a2b84561a5d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:34:40.204602 14523 raft_consensus.cc:729] T 4446e01173504e919b6fabfa38315ab2 P faaacac03e5747f48f17a2b84561a5d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faaacac03e5747f48f17a2b84561a5d8, State: Initialized, Role: FOLLOWER
I20260924 11:34:40.204826 14523 consensus_queue.cc:260] T 4446e01173504e919b6fabfa38315ab2 P faaacac03e5747f48f17a2b84561a5d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c42ff207f3406aa06f3984a03fc08b" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42317 } } peers { permanent_uuid: "faaacac03e5747f48f17a2b84561a5d8" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 34887 } } peers { permanent_uuid: "b1b300a0baf545cebef745d22579f93e" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 45033 } }
I20260924 11:34:40.205600 14526 tablet_bootstrap.cc:492] T 4446e01173504e919b6fabfa38315ab2 P b1b300a0baf545cebef745d22579f93e: No bootstrap required, opened a new log
I20260924 11:34:40.205675 14523 ts_tablet_manager.cc:1428] T 4446e01173504e919b6fabfa38315ab2 P faaacac03e5747f48f17a2b84561a5d8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 11:34:40.205817 14526 ts_tablet_manager.cc:1397] T 4446e01173504e919b6fabfa38315ab2 P b1b300a0baf545cebef745d22579f93e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.003s
I20260924 11:34:40.206530 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:40.206097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.6.212.2:44487)
0924 11:34:40.206295 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:40.206381 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260924 11:34:40.206849 14526 raft_consensus.cc:348] T 4446e01173504e919b6fabfa38315ab2 P b1b300a0baf545cebef745d22579f93e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c42ff207f3406aa06f3984a03fc08b" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42317 } } peers { permanent_uuid: "faaacac03e5747f48f17a2b84561a5d8" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 34887 } } peers { permanent_uuid: "b1b300a0baf545cebef745d22579f93e" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 45033 } }
I20260924 11:34:40.207175 14526 raft_consensus.cc:374] T 4446e01173504e919b6fabfa38315ab2 P b1b300a0baf545cebef745d22579f93e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:34:40.207386 14526 raft_consensus.cc:729] T 4446e01173504e919b6fabfa38315ab2 P b1b300a0baf545cebef745d22579f93e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1b300a0baf545cebef745d22579f93e, State: Initialized, Role: FOLLOWER
I20260924 11:34:40.207581 14274 heartbeater.cc:503] Master 127.6.212.61:38525 was elected leader, sending a full tablet report...
I20260924 11:34:40.207687 14526 consensus_queue.cc:260] T 4446e01173504e919b6fabfa38315ab2 P b1b300a0baf545cebef745d22579f93e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c42ff207f3406aa06f3984a03fc08b" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42317 } } peers { permanent_uuid: "faaacac03e5747f48f17a2b84561a5d8" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 34887 } } peers { permanent_uuid: "b1b300a0baf545cebef745d22579f93e" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 45033 } }
I20260924 11:34:40.208266 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:40.207179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.6.212.2:46217)
0924 11:34:40.208037 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:40.208117 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":758}
I20260924 11:34:40.208777 14526 ts_tablet_manager.cc:1428] T 4446e01173504e919b6fabfa38315ab2 P b1b300a0baf545cebef745d22579f93e: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260924 11:34:40.209877 14501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:40.209422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.6.212.3:59555)
0924 11:34:40.209595 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:40.209693 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 11:34:40.211158 14415 heartbeater.cc:503] Master 127.6.212.61:38525 was elected leader, sending a full tablet report...
I20260924 11:34:40.472432 14524 raft_consensus.cc:482] T 4446e01173504e919b6fabfa38315ab2 P 92c42ff207f3406aa06f3984a03fc08b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:34:40.472676 14524 raft_consensus.cc:504] T 4446e01173504e919b6fabfa38315ab2 P 92c42ff207f3406aa06f3984a03fc08b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c42ff207f3406aa06f3984a03fc08b" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42317 } } peers { permanent_uuid: "faaacac03e5747f48f17a2b84561a5d8" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 34887 } } peers { permanent_uuid: "b1b300a0baf545cebef745d22579f93e" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 45033 } }
I20260924 11:34:40.474241 14524 leader_election.cc:288] T 4446e01173504e919b6fabfa38315ab2 P 92c42ff207f3406aa06f3984a03fc08b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers faaacac03e5747f48f17a2b84561a5d8 (127.6.212.2:34887), b1b300a0baf545cebef745d22579f93e (127.6.212.3:45033)
I20260924 11:34:40.480372 14502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:40.474449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.2:34887 (local address 127.6.212.1:37103)
0924 11:34:40.474711 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:40.474736 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:40.474898 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:40.475284 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:40.475306 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:40.475354 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:34:40.475649 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:40.475661 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:40.476784 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:40.476790 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:40.478272 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:40.478283 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:40.479563 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:40.479580 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:40.479770 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:40.479805 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 11:34:40.479903 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 11:34:40.480049 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"spinlock_wait_cycles":4352}
I20260924 11:34:40.480556 14518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:40.474572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:37103 (local address 127.6.212.2:34887)
0924 11:34:40.474754 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:40.474761 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:40.474835 (+ 74us) server_negotiation.cc:407] Connection header received
0924 11:34:40.475072 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:40.475077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:40.475159 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:40.475261 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:34:40.475794 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:40.476651 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:40.478436 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:40.479340 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:40.479510 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:40.479587 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:40.480072 (+ 485us) server_negotiation.cc:299] Negotiation successful
0924 11:34:40.480199 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 11:34:40.480796 14520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:40.474736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:48417 (local address 127.6.212.3:45033)
0924 11:34:40.474900 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:40.474906 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:40.475332 (+ 426us) server_negotiation.cc:407] Connection header received
0924 11:34:40.475547 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:40.475553 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:40.475644 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:40.475742 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:34:40.476258 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:40.477155 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:40.478850 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:40.479775 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:40.479911 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:40.480003 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:40.480405 (+ 402us) server_negotiation.cc:299] Negotiation successful
0924 11:34:40.480566 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 11:34:40.481235 14530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:40.474496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.3:45033 (local address 127.6.212.1:48417)
0924 11:34:40.475227 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:40.475249 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:40.475393 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:40.475779 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:40.475783 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:40.475808 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:34:40.476120 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:40.476130 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:40.477281 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:40.477284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:40.478708 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:40.478721 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:40.479996 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:40.480005 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:40.480205 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:40.480236 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 11:34:40.480398 (+ 162us) client_negotiation.cc:240] Negotiation successful
0924 11:34:40.480516 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"spinlock_wait_cycles":896,"thread_start_us":382,"threads_started":1}
I20260924 11:34:40.481273 14227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4446e01173504e919b6fabfa38315ab2" candidate_uuid: "92c42ff207f3406aa06f3984a03fc08b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faaacac03e5747f48f17a2b84561a5d8" is_pre_election: true
I20260924 11:34:40.481566 14227 raft_consensus.cc:2454] T 4446e01173504e919b6fabfa38315ab2 P faaacac03e5747f48f17a2b84561a5d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92c42ff207f3406aa06f3984a03fc08b in term 0.
I20260924 11:34:40.481879 14368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4446e01173504e919b6fabfa38315ab2" candidate_uuid: "92c42ff207f3406aa06f3984a03fc08b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1b300a0baf545cebef745d22579f93e" is_pre_election: true
I20260924 11:34:40.482146 14368 raft_consensus.cc:2454] T 4446e01173504e919b6fabfa38315ab2 P b1b300a0baf545cebef745d22579f93e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92c42ff207f3406aa06f3984a03fc08b in term 0.
I20260924 11:34:40.482156 14015 leader_election.cc:302] T 4446e01173504e919b6fabfa38315ab2 P 92c42ff207f3406aa06f3984a03fc08b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92c42ff207f3406aa06f3984a03fc08b, faaacac03e5747f48f17a2b84561a5d8; no voters:
I20260924 11:34:40.482597 14524 raft_consensus.cc:2781] T 4446e01173504e919b6fabfa38315ab2 P 92c42ff207f3406aa06f3984a03fc08b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:34:40.482749 14524 raft_consensus.cc:482] T 4446e01173504e919b6fabfa38315ab2 P 92c42ff207f3406aa06f3984a03fc08b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:34:40.482851 14524 raft_consensus.cc:3037] T 4446e01173504e919b6fabfa38315ab2 P 92c42ff207f3406aa06f3984a03fc08b [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:34:40.484961 14524 raft_consensus.cc:504] T 4446e01173504e919b6fabfa38315ab2 P 92c42ff207f3406aa06f3984a03fc08b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c42ff207f3406aa06f3984a03fc08b" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42317 } } peers { permanent_uuid: "faaacac03e5747f48f17a2b84561a5d8" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 34887 } } peers { permanent_uuid: "b1b300a0baf545cebef745d22579f93e" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 45033 } }
I20260924 11:34:40.485565 14524 leader_election.cc:288] T 4446e01173504e919b6fabfa38315ab2 P 92c42ff207f3406aa06f3984a03fc08b [CANDIDATE]: Term 1 election: Requested vote from peers faaacac03e5747f48f17a2b84561a5d8 (127.6.212.2:34887), b1b300a0baf545cebef745d22579f93e (127.6.212.3:45033)
I20260924 11:34:40.485903 14227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4446e01173504e919b6fabfa38315ab2" candidate_uuid: "92c42ff207f3406aa06f3984a03fc08b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faaacac03e5747f48f17a2b84561a5d8"
I20260924 11:34:40.485920 14368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4446e01173504e919b6fabfa38315ab2" candidate_uuid: "92c42ff207f3406aa06f3984a03fc08b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1b300a0baf545cebef745d22579f93e"
I20260924 11:34:40.486099 14227 raft_consensus.cc:3037] T 4446e01173504e919b6fabfa38315ab2 P faaacac03e5747f48f17a2b84561a5d8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:34:40.486131 14368 raft_consensus.cc:3037] T 4446e01173504e919b6fabfa38315ab2 P b1b300a0baf545cebef745d22579f93e [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:34:40.488162 14227 raft_consensus.cc:2454] T 4446e01173504e919b6fabfa38315ab2 P faaacac03e5747f48f17a2b84561a5d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92c42ff207f3406aa06f3984a03fc08b in term 1.
I20260924 11:34:40.488324 14368 raft_consensus.cc:2454] T 4446e01173504e919b6fabfa38315ab2 P b1b300a0baf545cebef745d22579f93e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92c42ff207f3406aa06f3984a03fc08b in term 1.
I20260924 11:34:40.488562 14015 leader_election.cc:302] T 4446e01173504e919b6fabfa38315ab2 P 92c42ff207f3406aa06f3984a03fc08b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92c42ff207f3406aa06f3984a03fc08b, faaacac03e5747f48f17a2b84561a5d8; no voters:
I20260924 11:34:40.488814 14524 raft_consensus.cc:2781] T 4446e01173504e919b6fabfa38315ab2 P 92c42ff207f3406aa06f3984a03fc08b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:34:40.489274 14524 raft_consensus.cc:686] T 4446e01173504e919b6fabfa38315ab2 P 92c42ff207f3406aa06f3984a03fc08b [term 1 LEADER]: Becoming Leader. State: Replica: 92c42ff207f3406aa06f3984a03fc08b, State: Running, Role: LEADER
I20260924 11:34:40.489869 14524 consensus_queue.cc:237] T 4446e01173504e919b6fabfa38315ab2 P 92c42ff207f3406aa06f3984a03fc08b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c42ff207f3406aa06f3984a03fc08b" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42317 } } peers { permanent_uuid: "faaacac03e5747f48f17a2b84561a5d8" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 34887 } } peers { permanent_uuid: "b1b300a0baf545cebef745d22579f93e" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 45033 } }
I20260924 11:34:40.490187 14530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:40.489814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.6.212.1:34251)
0924 11:34:40.489958 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:40.490042 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 11:34:40.492543 14530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:40.492082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.6.212.1:60709)
0924 11:34:40.492278 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:40.492393 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260924 11:34:40.493551 13757 catalog_manager.cc:5495] T 4446e01173504e919b6fabfa38315ab2 P 92c42ff207f3406aa06f3984a03fc08b reported cstate change: term changed from 0 to 1, leader changed from <none> to 92c42ff207f3406aa06f3984a03fc08b (127.6.212.1). New cstate: current_term: 1 leader_uuid: "92c42ff207f3406aa06f3984a03fc08b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92c42ff207f3406aa06f3984a03fc08b" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42317 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "faaacac03e5747f48f17a2b84561a5d8" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 34887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1b300a0baf545cebef745d22579f93e" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 45033 } health_report { overall_health: UNKNOWN } } }
I20260924 11:34:40.549592 14516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:40.543945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46070 (local address 127.6.212.1:42317)
0924 11:34:40.544165 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:40.544171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:40.544192 (+ 21us) server_negotiation.cc:407] Connection header received
0924 11:34:40.544293 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:40.544299 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:40.544389 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:40.544531 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:34:40.544825 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:40.545870 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:40.547109 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:40.547768 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:40.547893 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:40.548279 (+ 386us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:34:40.548987 (+ 708us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:34:40.549154 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:40.549226 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 11:34:40.549303 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 11:34:40.553927 14368 raft_consensus.cc:1261] T 4446e01173504e919b6fabfa38315ab2 P b1b300a0baf545cebef745d22579f93e [term 1 FOLLOWER]: Refusing update from remote peer 92c42ff207f3406aa06f3984a03fc08b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:34:40.554327 14227 raft_consensus.cc:1261] T 4446e01173504e919b6fabfa38315ab2 P faaacac03e5747f48f17a2b84561a5d8 [term 1 FOLLOWER]: Refusing update from remote peer 92c42ff207f3406aa06f3984a03fc08b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:34:40.554608 14531 consensus_queue.cc:1035] T 4446e01173504e919b6fabfa38315ab2 P 92c42ff207f3406aa06f3984a03fc08b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1b300a0baf545cebef745d22579f93e" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 45033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:34:40.554997 14524 consensus_queue.cc:1035] T 4446e01173504e919b6fabfa38315ab2 P 92c42ff207f3406aa06f3984a03fc08b [LEADER]: Connected to new peer: Peer: permanent_uuid: "faaacac03e5747f48f17a2b84561a5d8" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 34887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:34:40.557858 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:40.557398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.6.212.2:44517)
0924 11:34:40.557609 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:40.557705 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260924 11:34:40.558068 14501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:40.557407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.6.212.3:45241)
0924 11:34:40.557730 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:40.557863 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177}
I20260924 11:34:40.563189 14534 mvcc.cc:205] Tried to move back new op lower bound from 7332862691541909504 to 7332862691290128384. Current Snapshot: MvccSnapshot[applied={T|T < 7332862691541909504}]
I20260924 11:34:40.564590 14536 mvcc.cc:205] Tried to move back new op lower bound from 7332862691541909504 to 7332862691290128384. Current Snapshot: MvccSnapshot[applied={T|T < 7332862691541909504}]
I20260924 11:34:40.565110 14501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:40.564407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.6.212.3:60713)
0924 11:34:40.564778 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:40.564895 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257}
I20260924 11:34:40.566143 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:40.565544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.6.212.2:59727)
0924 11:34:40.565821 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:40.565922 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
I20260924 11:34:40.572685 14530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:40.572229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.6.212.1:59929)
0924 11:34:40.572390 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:40.572510 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 11:34:40.574555 14530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:40.573852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.6.212.1:47175)
0924 11:34:40.574308 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:40.574403 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346}
I20260924 11:34:40.582854 13757 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:34:40.583194 13757 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:34:40.588228 13757 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d211d01c60f2417bb346495d00f5593f: Sending DeleteTablet for 3 replicas of tablet 4446e01173504e919b6fabfa38315ab2
I20260924 11:34:40.589253 14059 tablet_service.cc:1514] Processing DeleteTablet for tablet 4446e01173504e919b6fabfa38315ab2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:34:40 UTC) from {username='oryx', principal='oryx/127.6.212.61@KRBTEST.COM'} at 127.0.0.1:46058
I20260924 11:34:40.589370 14207 tablet_service.cc:1514] Processing DeleteTablet for tablet 4446e01173504e919b6fabfa38315ab2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:34:40 UTC) from {username='oryx', principal='oryx/127.6.212.61@KRBTEST.COM'} at 127.0.0.1:42116
I20260924 11:34:40.589745 14348 tablet_service.cc:1514] Processing DeleteTablet for tablet 4446e01173504e919b6fabfa38315ab2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:34:40 UTC) from {username='oryx', principal='oryx/127.6.212.61@KRBTEST.COM'} at 127.0.0.1:41880
I20260924 11:34:40.591307 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 13965
I20260924 11:34:40.600320 14519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:40.599843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.0.0.1:50854)
0924 11:34:40.600070 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:40.600161 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260924 11:34:40.600692 14545 tablet_replica.cc:330] T 4446e01173504e919b6fabfa38315ab2 P faaacac03e5747f48f17a2b84561a5d8: stopping tablet replica
I20260924 11:34:40.601065 14545 raft_consensus.cc:2229] T 4446e01173504e919b6fabfa38315ab2 P faaacac03e5747f48f17a2b84561a5d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:34:40.601460 14545 raft_consensus.cc:2258] T 4446e01173504e919b6fabfa38315ab2 P faaacac03e5747f48f17a2b84561a5d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:34:40.601552 14519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:40.601106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.0.0.1:50864)
0924 11:34:40.601288 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:40.601390 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260924 11:34:40.602169 14544 tablet_replica.cc:330] T 4446e01173504e919b6fabfa38315ab2 P 92c42ff207f3406aa06f3984a03fc08b: stopping tablet replica
I20260924 11:34:40.602526 14544 raft_consensus.cc:2229] T 4446e01173504e919b6fabfa38315ab2 P 92c42ff207f3406aa06f3984a03fc08b [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:34:40.603204 14545 ts_tablet_manager.cc:1905] T 4446e01173504e919b6fabfa38315ab2 P faaacac03e5747f48f17a2b84561a5d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:34:40.603380 14544 raft_consensus.cc:2258] T 4446e01173504e919b6fabfa38315ab2 P 92c42ff207f3406aa06f3984a03fc08b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:34:40.604707 14544 ts_tablet_manager.cc:1905] T 4446e01173504e919b6fabfa38315ab2 P 92c42ff207f3406aa06f3984a03fc08b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:34:40.614006 14544 ts_tablet_manager.cc:1918] T 4446e01173504e919b6fabfa38315ab2 P 92c42ff207f3406aa06f3984a03fc08b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:34:40.614243 14544 log.cc:1196] T 4446e01173504e919b6fabfa38315ab2 P 92c42ff207f3406aa06f3984a03fc08b: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-0/wal/wals/4446e01173504e919b6fabfa38315ab2
I20260924 11:34:40.614856 14544 ts_tablet_manager.cc:1939] T 4446e01173504e919b6fabfa38315ab2 P 92c42ff207f3406aa06f3984a03fc08b: Deleting consensus metadata
I20260924 11:34:40.614880 14546 tablet_replica.cc:330] T 4446e01173504e919b6fabfa38315ab2 P b1b300a0baf545cebef745d22579f93e: stopping tablet replica
I20260924 11:34:40.615089 14546 raft_consensus.cc:2229] T 4446e01173504e919b6fabfa38315ab2 P b1b300a0baf545cebef745d22579f93e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:34:40.615496 14546 raft_consensus.cc:2258] T 4446e01173504e919b6fabfa38315ab2 P b1b300a0baf545cebef745d22579f93e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:34:40.616253 13744 catalog_manager.cc:4841] TS 92c42ff207f3406aa06f3984a03fc08b (127.6.212.1:42317): tablet 4446e01173504e919b6fabfa38315ab2 (table test-table [id=dd04fe0f4bf34d73b602a5412b4079f0]) successfully deleted
I20260924 11:34:40.616786 14545 ts_tablet_manager.cc:1918] T 4446e01173504e919b6fabfa38315ab2 P faaacac03e5747f48f17a2b84561a5d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:34:40.617477 14546 ts_tablet_manager.cc:1905] T 4446e01173504e919b6fabfa38315ab2 P b1b300a0baf545cebef745d22579f93e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:34:40.623517 14545 log.cc:1196] T 4446e01173504e919b6fabfa38315ab2 P faaacac03e5747f48f17a2b84561a5d8: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-1/wal/wals/4446e01173504e919b6fabfa38315ab2
I20260924 11:34:40.623991 14546 ts_tablet_manager.cc:1918] T 4446e01173504e919b6fabfa38315ab2 P b1b300a0baf545cebef745d22579f93e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:34:40.624084 14545 ts_tablet_manager.cc:1939] T 4446e01173504e919b6fabfa38315ab2 P faaacac03e5747f48f17a2b84561a5d8: Deleting consensus metadata
I20260924 11:34:40.624161 14546 log.cc:1196] T 4446e01173504e919b6fabfa38315ab2 P b1b300a0baf545cebef745d22579f93e: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790249617057452-6992-0/minicluster-data/ts-2/wal/wals/4446e01173504e919b6fabfa38315ab2
I20260924 11:34:40.624629 14546 ts_tablet_manager.cc:1939] T 4446e01173504e919b6fabfa38315ab2 P b1b300a0baf545cebef745d22579f93e: Deleting consensus metadata
I20260924 11:34:40.625329 13744 catalog_manager.cc:4841] TS faaacac03e5747f48f17a2b84561a5d8 (127.6.212.2:34887): tablet 4446e01173504e919b6fabfa38315ab2 (table test-table [id=dd04fe0f4bf34d73b602a5412b4079f0]) successfully deleted
I20260924 11:34:40.625864 13742 catalog_manager.cc:4841] TS b1b300a0baf545cebef745d22579f93e (127.6.212.3:45033): tablet 4446e01173504e919b6fabfa38315ab2 (table test-table [id=dd04fe0f4bf34d73b602a5412b4079f0]) successfully deleted
I20260924 11:34:40.629477 14516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:40.592705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46074 (local address 127.6.212.1:42317)
0924 11:34:40.592879 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:40.592890 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:40.614524 (+ 21634us) server_negotiation.cc:407] Connection header received
0924 11:34:40.614704 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:40.614710 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:40.614807 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:40.614934 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:40.617103 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:40.618070 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:40.619471 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:40.619915 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:40.620032 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:40.622635 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:40.622657 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:40.622672 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:40.622709 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:40.626653 (+ 3944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:40.627283 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:40.627291 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:40.627295 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:40.627519 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:40.627985 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:40.627989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:40.627991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:40.628399 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:40.628678 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:40.629023 (+ 345us) server_negotiation.cc:299] Negotiation successful
0924 11:34:40.629167 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 11:34:40.630096 14119 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:34:40.733309 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 13965
I20260924 11:34:40.752029 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 14136
I20260924 11:34:40.764550 14518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:40.752711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42128 (local address 127.6.212.2:34887)
0924 11:34:40.752898 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:40.752902 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:40.752920 (+ 18us) server_negotiation.cc:407] Connection header received
0924 11:34:40.753069 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:40.753073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:40.753132 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:40.753225 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:40.754394 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:40.755080 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:40.756397 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:40.756786 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:40.756879 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:40.759175 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:40.759188 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:40.759203 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:40.759235 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:40.762045 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:40.762618 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:40.762624 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:40.762626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:40.762768 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:40.763116 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:40.763119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:40.763121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:40.763535 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:40.763750 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:40.764061 (+ 311us) server_negotiation.cc:299] Negotiation successful
0924 11:34:40.764209 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 11:34:40.765172 14267 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:34:40.883777 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 14136
I20260924 11:34:40.906452 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 14285
I20260924 11:34:40.919333 14520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:40.907185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41882 (local address 127.6.212.3:45033)
0924 11:34:40.907342 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:40.907346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:40.907461 (+ 115us) server_negotiation.cc:407] Connection header received
0924 11:34:40.907685 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:40.907690 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:40.907767 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:40.907876 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:40.909107 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:40.909805 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:40.910921 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:40.911319 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:40.911490 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:40.913826 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:40.913852 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:40.913867 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:40.913895 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:40.916730 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:40.917331 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:40.917338 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:40.917341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:40.917458 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:40.917943 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:40.917945 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:40.917947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:40.918281 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:40.918524 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:40.918843 (+ 319us) server_negotiation.cc:299] Negotiation successful
0924 11:34:40.919004 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 11:34:40.920027 14408 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:34:41.012933 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 14285
I20260924 11:34:41.021939 14519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:41.021525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.0.0.1:50878)
0924 11:34:41.021706 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:41.021793 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260924 11:34:41.023058 14519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:41.022719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:42625 (local address 127.0.0.1:50884)
0924 11:34:41.022871 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:41.022934 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:42625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 11:34:41.030992 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 13620
I20260924 11:34:41.032109 13787 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:34:41.153201 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 13620
W20260924 11:34:41.174438 13930 connection.cc:535] server connection from 127.0.0.1:52944 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 11:34:41.174813 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 13720
I20260924 11:34:41.175884 13974 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:34:41.289166 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 13720
2026-09-24T11:34:41Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (11765 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:34:41 dist-test-slave-cq7c krb5kdc[14555](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:34:41 dist-test-slave-cq7c krb5kdc[14555](info): set up 2 sockets
Sep 24 11:34:41 dist-test-slave-cq7c krb5kdc[14555](info): commencing operation
krb5kdc: starting...
W20260924 11:34:42.396226 6992 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.004s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Sep 24 11:34:42 dist-test-slave-cq7c krb5kdc[14555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:34:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:34:42Z Disabled control of system clock
WARNING: no policy specified for oryx/127.6.212.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.212.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.212.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.62.keytab.
Entry for principal oryx/127.6.212.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.62.keytab.
I20260924 11:34:42.619089 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:39467
--webserver_interface=127.6.212.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.62.keytab
--principal=oryx/127.6.212.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:45745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:39467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:34:42.937615 14568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:42.937896 14568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:42.938077 14568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:42.945878 14568 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:34:42.946069 14568 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 11:34:42.946163 14568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:42.946292 14568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:42.946400 14568 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:34:42.946534 14568 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:34:42.957691 14568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:45745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:39467
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.62.keytab
--principal=oryx/127.6.212.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:39467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.212.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:34:42.959051 14568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:42.960474 14568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:42.973101 14576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:42.973147 14573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:42.973181 14574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:42.973361 14568 server_base.cc:1034] running on GCE node
I20260924 11:34:42.974429 14568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:42.975857 14568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:42.977064 14568 hybrid_clock.cc:648] HybridClock initialized: now 1790249682977009 us; error 27 us; skew 500 ppm
I20260924 11:34:42.977389 14568 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:34:42 dist-test-slave-cq7c krb5kdc[14555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249682, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.212.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:42.980549 14568 init.cc:377] Logged in from keytab as oryx/127.6.212.62@KRBTEST.COM (short username oryx)
I20260924 11:34:42.981984 14568 webserver.cc:460] Webserver started at http://127.6.212.62:42239/ using document root <none> and password file <none>
I20260924 11:34:42.982625 14568 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:42.982709 14568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:42.982966 14568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:34:42.985970 14568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "3c9bf26035dc4b5e9edde52527a113e7"
format_stamp: "Formatted at 2026-09-24 11:34:42 on dist-test-slave-cq7c"
server_key: "81aa83029c55bc9879370e875a9d7a68"
server_key_iv: "d2849d11e9c1074ccb83adcb0a75ca1b"
server_key_version: "encryptionkey@0"
I20260924 11:34:42.986696 14568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "3c9bf26035dc4b5e9edde52527a113e7"
format_stamp: "Formatted at 2026-09-24 11:34:42 on dist-test-slave-cq7c"
server_key: "81aa83029c55bc9879370e875a9d7a68"
server_key_iv: "d2849d11e9c1074ccb83adcb0a75ca1b"
server_key_version: "encryptionkey@0"
I20260924 11:34:42.991708 14568 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260924 11:34:42.994979 14583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:42.996376 14568 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:34:42.996511 14568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "3c9bf26035dc4b5e9edde52527a113e7"
format_stamp: "Formatted at 2026-09-24 11:34:42 on dist-test-slave-cq7c"
server_key: "81aa83029c55bc9879370e875a9d7a68"
server_key_iv: "d2849d11e9c1074ccb83adcb0a75ca1b"
server_key_version: "encryptionkey@0"
I20260924 11:34:42.996680 14568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:43.011699 14568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:43.015645 14568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:34:43.015865 14568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:43.037825 14568 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.62:39467
I20260924 11:34:43.037942 14644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.62:39467 every 8 connection(s)
I20260924 11:34:43.039227 14568 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:34:43.039729 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 14568
I20260924 11:34:43.040010 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:34:43.040941 6992 external_mini_cluster.cc:1428] Setting key ab80a928b67f96b2531d24ad70b75042
I20260924 11:34:43.043996 14645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 11:34:43 dist-test-slave-cq7c krb5kdc[14555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM
I20260924 11:34:43.056869 14645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c9bf26035dc4b5e9edde52527a113e7: Bootstrap starting.
I20260924 11:34:43.060206 14645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c9bf26035dc4b5e9edde52527a113e7: Neither blocks nor log segments found. Creating new log.
I20260924 11:34:43.061270 14645 log.cc:824] T 00000000000000000000000000000000 P 3c9bf26035dc4b5e9edde52527a113e7: Log is configured to *not* fsync() on all Append() calls
I20260924 11:34:43.063231 14648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:43.042986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43376 (local address 127.6.212.62:39467)
0924 11:34:43.043763 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:43.043775 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:43.043814 (+ 39us) server_negotiation.cc:407] Connection header received
0924 11:34:43.045420 (+ 1606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:43.045441 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:43.045801 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:43.046119 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:43.047814 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:43.049145 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:43.050918 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:43.051619 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:43.051758 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:43.056342 (+ 4584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:43.056350 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:43.056364 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:43.056403 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:43.059270 (+ 2867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:43.059956 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:43.059963 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:43.059968 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:43.060126 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:43.060569 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:43.060574 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:43.060579 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:43.061085 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:43.061374 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:43.061868 (+ 494us) server_negotiation.cc:299] Negotiation successful
0924 11:34:43.062284 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":382,"threads_started":1}
I20260924 11:34:43.064445 14645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c9bf26035dc4b5e9edde52527a113e7: No bootstrap required, opened a new log
I20260924 11:34:43.072214 14645 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c9bf26035dc4b5e9edde52527a113e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9bf26035dc4b5e9edde52527a113e7" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 39467 } }
I20260924 11:34:43.072569 14645 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c9bf26035dc4b5e9edde52527a113e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:34:43.072642 14645 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c9bf26035dc4b5e9edde52527a113e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c9bf26035dc4b5e9edde52527a113e7, State: Initialized, Role: FOLLOWER
I20260924 11:34:43.073129 14645 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c9bf26035dc4b5e9edde52527a113e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9bf26035dc4b5e9edde52527a113e7" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 39467 } }
I20260924 11:34:43.073321 14645 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c9bf26035dc4b5e9edde52527a113e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:34:43.073477 14645 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c9bf26035dc4b5e9edde52527a113e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:34:43.073602 14645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c9bf26035dc4b5e9edde52527a113e7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:34:43.076030 14645 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c9bf26035dc4b5e9edde52527a113e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9bf26035dc4b5e9edde52527a113e7" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 39467 } }
I20260924 11:34:43.076470 14645 leader_election.cc:302] T 00000000000000000000000000000000 P 3c9bf26035dc4b5e9edde52527a113e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c9bf26035dc4b5e9edde52527a113e7; no voters:
I20260924 11:34:43.076977 14645 leader_election.cc:288] T 00000000000000000000000000000000 P 3c9bf26035dc4b5e9edde52527a113e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:34:43.077133 14650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c9bf26035dc4b5e9edde52527a113e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:34:43.077818 14650 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c9bf26035dc4b5e9edde52527a113e7 [term 1 LEADER]: Becoming Leader. State: Replica: 3c9bf26035dc4b5e9edde52527a113e7, State: Running, Role: LEADER
I20260924 11:34:43.078312 14650 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c9bf26035dc4b5e9edde52527a113e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9bf26035dc4b5e9edde52527a113e7" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 39467 } }
I20260924 11:34:43.079026 14645 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c9bf26035dc4b5e9edde52527a113e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:34:43.080979 14651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c9bf26035dc4b5e9edde52527a113e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c9bf26035dc4b5e9edde52527a113e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9bf26035dc4b5e9edde52527a113e7" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 39467 } } }
I20260924 11:34:43.081331 14651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c9bf26035dc4b5e9edde52527a113e7 [sys.catalog]: This master's current role is: LEADER
I20260924 11:34:43.081768 14652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c9bf26035dc4b5e9edde52527a113e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c9bf26035dc4b5e9edde52527a113e7. Latest consensus state: current_term: 1 leader_uuid: "3c9bf26035dc4b5e9edde52527a113e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9bf26035dc4b5e9edde52527a113e7" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 39467 } } }
I20260924 11:34:43.082006 14652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c9bf26035dc4b5e9edde52527a113e7 [sys.catalog]: This master's current role is: LEADER
I20260924 11:34:43.084033 14656 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:34:43.089699 14656 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:34:43.098691 14656 catalog_manager.cc:1347] Generated new cluster ID: bd082a616c2f4828bbaf6bd28ddd0347
I20260924 11:34:43.098803 14656 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:34:43.125982 14656 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:34:43.128010 14656 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:34:43.135828 14656 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c9bf26035dc4b5e9edde52527a113e7: Generated new TSK 0
I20260924 11:34:43.136657 14656 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.6.212.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.212.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.212.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.1.keytab.
Entry for principal oryx/127.6.212.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.1.keytab.
I20260924 11:34:43.199743 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.1:0
--local_ip_for_outbound_sockets=127.6.212.1
--webserver_interface=127.6.212.1
--webserver_port=0
--tserver_master_addrs=127.6.212.62:39467
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.1.keytab
--principal=oryx/127.6.212.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:45745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:34:43.529735 14671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:43.530004 14671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:43.530172 14671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:43.537452 14671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:43.537556 14671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:43.537664 14671 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 11:34:43.537818 14671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.1
I20260924 11:34:43.547353 14671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:45745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.1.keytab
--principal=oryx/127.6.212.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.212.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.212.62:39467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.1
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:34:43.548631 14671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:43.549944 14671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:43.563045 14676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:43.563261 14677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:43.563293 14679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:43.569228 14671 server_base.cc:1034] running on GCE node
I20260924 11:34:43.569893 14671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:43.570865 14671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:43.572121 14671 hybrid_clock.cc:648] HybridClock initialized: now 1790249683572064 us; error 46 us; skew 500 ppm
I20260924 11:34:43.572434 14671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:34:43 dist-test-slave-cq7c krb5kdc[14555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249683, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.212.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:43.575585 14671 init.cc:377] Logged in from keytab as oryx/127.6.212.1@KRBTEST.COM (short username oryx)
I20260924 11:34:43.577039 14671 webserver.cc:460] Webserver started at http://127.6.212.1:44217/ using document root <none> and password file <none>
I20260924 11:34:43.577692 14671 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:43.577773 14671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:43.578042 14671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:34:43.581094 14671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-0/data/instance:
uuid: "4dddd225a8d649fabae7c12fa9fe99cc"
format_stamp: "Formatted at 2026-09-24 11:34:43 on dist-test-slave-cq7c"
server_key: "70a61235bf16c7ab6c11b5eab8c99bcb"
server_key_iv: "0460963aa4351f1d135fe165d8625c16"
server_key_version: "encryptionkey@0"
I20260924 11:34:43.581812 14671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance:
uuid: "4dddd225a8d649fabae7c12fa9fe99cc"
format_stamp: "Formatted at 2026-09-24 11:34:43 on dist-test-slave-cq7c"
server_key: "70a61235bf16c7ab6c11b5eab8c99bcb"
server_key_iv: "0460963aa4351f1d135fe165d8625c16"
server_key_version: "encryptionkey@0"
I20260924 11:34:43.587159 14671 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260924 11:34:43.590587 14686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:43.591965 14671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:34:43.592092 14671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-0/wal
uuid: "4dddd225a8d649fabae7c12fa9fe99cc"
format_stamp: "Formatted at 2026-09-24 11:34:43 on dist-test-slave-cq7c"
server_key: "70a61235bf16c7ab6c11b5eab8c99bcb"
server_key_iv: "0460963aa4351f1d135fe165d8625c16"
server_key_version: "encryptionkey@0"
I20260924 11:34:43.592268 14671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:43.625370 14671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:43.629441 14671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:34:43.629657 14671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:43.632392 14671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:34:43.632475 14671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:43.632678 14671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:34:43.632738 14671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 11:34:43 dist-test-slave-cq7c krb5kdc[14555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249683, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.212.1@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM
I20260924 11:34:43.657768 14695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:43.633127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:48895 (local address 127.6.212.62:39467)
0924 11:34:43.633710 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:43.633719 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:43.635100 (+ 1381us) server_negotiation.cc:407] Connection header received
0924 11:34:43.638320 (+ 3220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:43.638329 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:43.638425 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:43.638540 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:43.641589 (+ 3049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:43.642544 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:43.645791 (+ 3247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:43.646364 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:43.646515 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:43.651346 (+ 4831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:43.651351 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:43.651395 (+ 44us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:43.651437 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:43.654727 (+ 3290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:43.655508 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:43.655514 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:43.655517 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:43.655645 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:43.656089 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:43.656099 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:43.656101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:43.656346 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:43.656494 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:43.657212 (+ 718us) server_negotiation.cc:299] Negotiation successful
0924 11:34:43.657425 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":320,"threads_started":1}
I20260924 11:34:43.659111 14696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:43.634023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:39467 (local address 127.6.212.1:48895)
0924 11:34:43.634916 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:43.634975 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:43.637928 (+ 2953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:43.639158 (+ 1230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:43.639175 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:43.640265 (+ 1090us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:43.641323 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:43.641356 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:43.642833 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:43.642843 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:43.645616 (+ 2773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:43.645635 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:43.646656 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:43.646665 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:43.647034 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:43.647972 (+ 938us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:43.647996 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:43.651122 (+ 3126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:43.654892 (+ 3770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:43.654901 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:43.654915 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:43.655339 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:43.655767 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:43.655772 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:43.655776 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:43.655970 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:43.656466 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:43.656475 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:43.656846 (+ 371us) client_negotiation.cc:769] Sending connection context
0924 11:34:43.657109 (+ 263us) client_negotiation.cc:240] Negotiation successful
0924 11:34:43.657636 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":364,"threads_started":1}
I20260924 11:34:43.689878 14671 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.1:42819
I20260924 11:34:43.689929 14806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.1:42819 every 8 connection(s)
I20260924 11:34:43.691229 14671 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
I20260924 11:34:43.691382 14695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:43.675009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:59809 (local address 127.6.212.62:39467)
0924 11:34:43.675205 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:43.675212 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:43.675986 (+ 774us) server_negotiation.cc:407] Connection header received
0924 11:34:43.676200 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:43.676208 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:43.676320 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:43.676513 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:43.677375 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:43.678832 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:43.680510 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:43.681260 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:43.681416 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:43.684732 (+ 3316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:43.684741 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:43.684749 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:43.684799 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:43.688379 (+ 3580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:43.689320 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:43.689327 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:43.689333 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:43.689488 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:43.689849 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:43.689855 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:43.689858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:43.690226 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:43.690375 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:43.690764 (+ 389us) server_negotiation.cc:299] Negotiation successful
0924 11:34:43.690988 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 11:34:43.691509 14780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:43.675108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:39467 (local address 127.6.212.1:59809)
0924 11:34:43.675857 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:43.675882 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:43.676052 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:43.676526 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:43.676529 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:43.676954 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:43.677232 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:43.677239 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:43.678991 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:43.678994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:43.680322 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:43.680336 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:43.681726 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:43.681732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:43.682012 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:43.682866 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:43.682899 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:43.684473 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:43.688524 (+ 4051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:43.688531 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:43.688538 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:43.689173 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:43.689589 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:43.689599 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:43.689603 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:43.689728 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:43.690339 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:43.690343 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:43.690532 (+ 189us) client_negotiation.cc:769] Sending connection context
0924 11:34:43.690666 (+ 134us) client_negotiation.cc:240] Negotiation successful
0924 11:34:43.690999 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":444,"threads_started":1}
I20260924 11:34:43.693755 14807 heartbeater.cc:347] Connected to a master server at 127.6.212.62:39467
I20260924 11:34:43.694012 14807 heartbeater.cc:464] Registering TS with master...
I20260924 11:34:43.694813 14807 heartbeater.cc:511] Master 127.6.212.62:39467 requested a full tablet report, sending...
I20260924 11:34:43.695312 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 14671
I20260924 11:34:43.695564 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance
I20260924 11:34:43.696313 6992 external_mini_cluster.cc:1428] Setting key 5a8c381f953ced81463b9fc092e3b1e1
I20260924 11:34:43.697371 14600 ts_manager.cc:194] Registered new tserver with Master: 4dddd225a8d649fabae7c12fa9fe99cc (127.6.212.1:42819)
I20260924 11:34:43.699121 14600 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.212.1@KRBTEST.COM'} at 127.6.212.1:48895
WARNING: no policy specified for oryx/127.6.212.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.212.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.212.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.2.keytab.
Entry for principal oryx/127.6.212.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.2.keytab.
I20260924 11:34:43.754333 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.2:0
--local_ip_for_outbound_sockets=127.6.212.2
--webserver_interface=127.6.212.2
--webserver_port=0
--tserver_master_addrs=127.6.212.62:39467
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.2.keytab
--principal=oryx/127.6.212.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:45745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:34:44.075865 14811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:44.076141 14811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:44.076306 14811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:44.083922 14811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:44.084041 14811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:44.084193 14811 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 11:34:44.084331 14811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.2
I20260924 11:34:44.094365 14811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:45745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.2.keytab
--principal=oryx/127.6.212.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.212.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.212.62:39467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.2
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:34:44.095745 14811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:44.097110 14811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:44.110641 14816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:44.110697 14817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:44.110671 14819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:44.111441 14811 server_base.cc:1034] running on GCE node
I20260924 11:34:44.112054 14811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:44.113023 14811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:44.114280 14811 hybrid_clock.cc:648] HybridClock initialized: now 1790249684114207 us; error 38 us; skew 500 ppm
I20260924 11:34:44.114585 14811 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:34:44 dist-test-slave-cq7c krb5kdc[14555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249684, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.212.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:44.117653 14811 init.cc:377] Logged in from keytab as oryx/127.6.212.2@KRBTEST.COM (short username oryx)
I20260924 11:34:44.119010 14811 webserver.cc:460] Webserver started at http://127.6.212.2:43915/ using document root <none> and password file <none>
I20260924 11:34:44.119752 14811 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:44.119884 14811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:44.120189 14811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:34:44.122963 14811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-1/data/instance:
uuid: "432ac21283b94a269b47bdb8f90040b0"
format_stamp: "Formatted at 2026-09-24 11:34:44 on dist-test-slave-cq7c"
server_key: "d29840adb4907f1ad2f6719a236a4319"
server_key_iv: "1c72a79852a27476f784a2149ba1817a"
server_key_version: "encryptionkey@0"
I20260924 11:34:44.123746 14811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance:
uuid: "432ac21283b94a269b47bdb8f90040b0"
format_stamp: "Formatted at 2026-09-24 11:34:44 on dist-test-slave-cq7c"
server_key: "d29840adb4907f1ad2f6719a236a4319"
server_key_iv: "1c72a79852a27476f784a2149ba1817a"
server_key_version: "encryptionkey@0"
I20260924 11:34:44.128705 14811 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260924 11:34:44.132078 14826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:44.133387 14811 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:34:44.133551 14811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-1/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-1/wal
uuid: "432ac21283b94a269b47bdb8f90040b0"
format_stamp: "Formatted at 2026-09-24 11:34:44 on dist-test-slave-cq7c"
server_key: "d29840adb4907f1ad2f6719a236a4319"
server_key_iv: "1c72a79852a27476f784a2149ba1817a"
server_key_version: "encryptionkey@0"
I20260924 11:34:44.133785 14811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:44.150171 14811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:44.156070 14811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:34:44.156359 14811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:44.160180 14811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:34:44.160292 14811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:44.160447 14811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:34:44.160523 14811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 11:34:44 dist-test-slave-cq7c krb5kdc[14555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249684, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.212.2@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM
I20260924 11:34:44.186086 14695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:44.161124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:57143 (local address 127.6.212.62:39467)
0924 11:34:44.161347 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:44.161354 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:44.162954 (+ 1600us) server_negotiation.cc:407] Connection header received
0924 11:34:44.165674 (+ 2720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:44.165683 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:44.165789 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:44.165905 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:44.167418 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:44.168693 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:44.171415 (+ 2722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:44.172001 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:44.172137 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:44.178353 (+ 6216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:44.178364 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:44.178370 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:44.178420 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:44.181945 (+ 3525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:44.183155 (+ 1210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:44.183161 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:44.183165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:44.183332 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:44.184091 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:44.184095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:44.184098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:44.184391 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:44.184524 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:44.185657 (+ 1133us) server_negotiation.cc:299] Negotiation successful
0924 11:34:44.185824 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 11:34:44.188530 14835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:44.161790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:39467 (local address 127.6.212.2:57143)
0924 11:34:44.162807 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:44.162844 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:44.165251 (+ 2407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:44.166118 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:44.166125 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:44.166650 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:44.167223 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:44.167240 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:44.168861 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:44.168869 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:44.171154 (+ 2285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:44.171170 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:44.172464 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:44.172473 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:44.172792 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:44.173883 (+ 1091us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:44.173914 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:44.177974 (+ 4060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:44.182185 (+ 4211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:44.182200 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:44.182223 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:44.183005 (+ 782us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:44.183563 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:44.183571 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:44.183578 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:44.183960 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:44.184548 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:44.184560 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:44.185160 (+ 600us) client_negotiation.cc:769] Sending connection context
0924 11:34:44.185556 (+ 396us) client_negotiation.cc:240] Negotiation successful
0924 11:34:44.186294 (+ 738us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"thread_start_us":434,"threads_started":1}
I20260924 11:34:44.204470 14811 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.2:37311
I20260924 11:34:44.204470 14941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.2:37311 every 8 connection(s)
I20260924 11:34:44.205789 14811 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
I20260924 11:34:44.208349 14943 heartbeater.cc:347] Connected to a master server at 127.6.212.62:39467
I20260924 11:34:44.208554 14943 heartbeater.cc:464] Registering TS with master...
I20260924 11:34:44.209302 14943 heartbeater.cc:511] Master 127.6.212.62:39467 requested a full tablet report, sending...
I20260924 11:34:44.210692 14600 ts_manager.cc:194] Registered new tserver with Master: 432ac21283b94a269b47bdb8f90040b0 (127.6.212.2:37311)
I20260924 11:34:44.211544 14600 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.212.2@KRBTEST.COM'} at 127.6.212.2:57143
I20260924 11:34:44.214462 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 14811
I20260924 11:34:44.214658 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance
I20260924 11:34:44.215229 6992 external_mini_cluster.cc:1428] Setting key f8b26a879eba5530f8dc5bb009406933
I20260924 11:34:44.222195 14947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:44.209585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:39467 (local address 127.6.212.2:60013)
0924 11:34:44.210212 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:44.210230 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:44.210386 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:44.210709 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:44.210713 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:44.211149 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:44.211420 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:44.211432 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:44.212593 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:44.212597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:44.213650 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:44.213661 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:44.214474 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:44.214480 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:44.214657 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:44.215300 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:44.215316 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:44.216207 (+ 891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:44.219912 (+ 3705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:44.219915 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:44.219918 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:44.220272 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:44.220655 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:44.220658 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:44.220660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:44.220769 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:44.221426 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:44.221428 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:44.221582 (+ 154us) client_negotiation.cc:769] Sending connection context
0924 11:34:44.221689 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 11:34:44.221887 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"mutex_wait_us":2,"thread_start_us":390,"threads_started":1}
I20260924 11:34:44.222615 14695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:44.209911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:60013 (local address 127.6.212.62:39467)
0924 11:34:44.210119 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:44.210126 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:44.210322 (+ 196us) server_negotiation.cc:407] Connection header received
0924 11:34:44.210525 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:44.210529 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:44.210589 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:44.210695 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:44.211563 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:44.212457 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:44.213789 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:44.214269 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:44.214434 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:44.216390 (+ 1956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:44.216395 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:44.216398 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:44.216425 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:44.219792 (+ 3367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:44.220393 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:44.220401 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:44.220407 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:44.220552 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:44.220865 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:44.220872 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:44.220878 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:44.221308 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:44.221465 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:44.221817 (+ 352us) server_negotiation.cc:299] Negotiation successful
0924 11:34:44.222044 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
WARNING: no policy specified for oryx/127.6.212.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.212.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.212.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.3.keytab.
Entry for principal oryx/127.6.212.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.3.keytab.
I20260924 11:34:44.267699 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.3:0
--local_ip_for_outbound_sockets=127.6.212.3
--webserver_interface=127.6.212.3
--webserver_port=0
--tserver_master_addrs=127.6.212.62:39467
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.3.keytab
--principal=oryx/127.6.212.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:45745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:34:44.589439 14950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:44.589709 14950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:44.589890 14950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:44.597378 14950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:44.597482 14950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:44.597662 14950 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 11:34:44.597752 14950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.3
I20260924 11:34:44.607462 14950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:45745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.3.keytab
--principal=oryx/127.6.212.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.212.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.212.62:39467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.3
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:34:44.608711 14950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:44.610044 14950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:44.623266 14956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:44.623521 14955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:44.623692 14958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:44.624239 14950 server_base.cc:1034] running on GCE node
I20260924 11:34:44.624884 14950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:44.625749 14950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:44.626968 14950 hybrid_clock.cc:648] HybridClock initialized: now 1790249684626891 us; error 37 us; skew 500 ppm
I20260924 11:34:44.627235 14950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:34:44 dist-test-slave-cq7c krb5kdc[14555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249684, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.212.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:44.634673 14950 init.cc:377] Logged in from keytab as oryx/127.6.212.3@KRBTEST.COM (short username oryx)
I20260924 11:34:44.636198 14950 webserver.cc:460] Webserver started at http://127.6.212.3:36003/ using document root <none> and password file <none>
I20260924 11:34:44.637007 14950 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:44.637115 14950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:44.637451 14950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:34:44.640363 14950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-2/data/instance:
uuid: "2a86a28876eb407596a0452c86e6dd45"
format_stamp: "Formatted at 2026-09-24 11:34:44 on dist-test-slave-cq7c"
server_key: "9a50442f5bd7221943b285e7918c70f2"
server_key_iv: "f7a5af0381f2eda4cc64879730ad3b67"
server_key_version: "encryptionkey@0"
I20260924 11:34:44.641077 14950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance:
uuid: "2a86a28876eb407596a0452c86e6dd45"
format_stamp: "Formatted at 2026-09-24 11:34:44 on dist-test-slave-cq7c"
server_key: "9a50442f5bd7221943b285e7918c70f2"
server_key_iv: "f7a5af0381f2eda4cc64879730ad3b67"
server_key_version: "encryptionkey@0"
I20260924 11:34:44.645951 14950 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 11:34:44.649376 14965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:44.650674 14950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 11:34:44.650832 14950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-2/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-2/wal
uuid: "2a86a28876eb407596a0452c86e6dd45"
format_stamp: "Formatted at 2026-09-24 11:34:44 on dist-test-slave-cq7c"
server_key: "9a50442f5bd7221943b285e7918c70f2"
server_key_iv: "f7a5af0381f2eda4cc64879730ad3b67"
server_key_version: "encryptionkey@0"
I20260924 11:34:44.651055 14950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:44.667732 14950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:44.671957 14950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:34:44.672223 14950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:44.674679 14950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:34:44.674788 14950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:44.674958 14950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:34:44.675034 14950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 11:34:44 dist-test-slave-cq7c krb5kdc[14555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249684, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.212.3@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM
I20260924 11:34:44.697959 14695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:44.675419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:39085 (local address 127.6.212.62:39467)
0924 11:34:44.675612 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:44.675619 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:44.677048 (+ 1429us) server_negotiation.cc:407] Connection header received
0924 11:34:44.679179 (+ 2131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:44.679188 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:44.679292 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:44.679500 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:44.681768 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:44.682834 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:44.685201 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:44.685655 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:44.685804 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:44.690301 (+ 4497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:44.690308 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:44.690312 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:44.690355 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:44.693885 (+ 3530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:44.694758 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:44.694763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:44.694766 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:44.694925 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:44.695578 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:44.695583 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:44.695587 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:44.695925 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:44.696078 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:44.697180 (+ 1102us) server_negotiation.cc:299] Negotiation successful
0924 11:34:44.697488 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 11:34:44.699522 14974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:44.675989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:39467 (local address 127.6.212.3:39085)
0924 11:34:44.676851 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:44.676896 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:44.678782 (+ 1886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:44.679789 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:44.679797 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:44.680616 (+ 819us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:44.681561 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:44.681595 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:44.682998 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:44.683003 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:44.685041 (+ 2038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:44.685060 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:44.685880 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:44.685889 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:44.686124 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:44.686855 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:44.686881 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:44.690012 (+ 3131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:44.694044 (+ 4032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:44.694053 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:44.694069 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:44.694617 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:44.695063 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:44.695068 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:44.695071 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:44.695427 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:44.696075 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:44.696087 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:44.696647 (+ 560us) client_negotiation.cc:769] Sending connection context
0924 11:34:44.697011 (+ 364us) client_negotiation.cc:240] Negotiation successful
0924 11:34:44.697649 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":363,"threads_started":1}
I20260924 11:34:44.702544 14807 heartbeater.cc:503] Master 127.6.212.62:39467 was elected leader, sending a full tablet report...
I20260924 11:34:44.723237 14950 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.3:41957
I20260924 11:34:44.723285 15084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.3:41957 every 8 connection(s)
I20260924 11:34:44.724504 14950 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
I20260924 11:34:44.724736 14695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:44.708781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:56595 (local address 127.6.212.62:39467)
0924 11:34:44.708957 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:44.708964 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:44.709835 (+ 871us) server_negotiation.cc:407] Connection header received
0924 11:34:44.710049 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:44.710057 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:44.710155 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:44.710269 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:44.712052 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:44.712910 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:44.714654 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:44.715181 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:44.715314 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:44.718107 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:44.718113 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:44.718119 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:44.718155 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:44.721537 (+ 3382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:44.722369 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:44.722374 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:44.722377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:44.722785 (+ 408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:44.723199 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:44.723206 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:44.723210 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:44.723589 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:44.723740 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:44.724140 (+ 400us) server_negotiation.cc:299] Negotiation successful
0924 11:34:44.724365 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 11:34:44.724869 15045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:44.708852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:39467 (local address 127.6.212.3:56595)
0924 11:34:44.709708 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:44.709733 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:44.709901 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:44.710496 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:44.710503 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:44.711308 (+ 805us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:44.711887 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:44.711905 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:44.713116 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:44.713123 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:44.714516 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:44.714531 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:44.715530 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:44.715535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:44.715787 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:44.716504 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:44.716532 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:44.717874 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:44.721666 (+ 3792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:44.721671 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:44.721676 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:44.722216 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:44.722925 (+ 709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:44.722930 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:44.722933 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:44.723070 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:44.723710 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:44.723715 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:44.723899 (+ 184us) client_negotiation.cc:769] Sending connection context
0924 11:34:44.724043 (+ 144us) client_negotiation.cc:240] Negotiation successful
0924 11:34:44.724386 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"thread_start_us":472,"threads_started":1}
I20260924 11:34:44.728528 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 14950
I20260924 11:34:44.728710 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance
I20260924 11:34:44.729210 6992 external_mini_cluster.cc:1428] Setting key b07a6e0571fd08336998afcdbba65ad8
I20260924 11:34:44.727722 15085 heartbeater.cc:347] Connected to a master server at 127.6.212.62:39467
I20260924 11:34:44.730043 15085 heartbeater.cc:464] Registering TS with master...
I20260924 11:34:44.730783 15085 heartbeater.cc:511] Master 127.6.212.62:39467 requested a full tablet report, sending...
I20260924 11:34:44.732105 14599 ts_manager.cc:194] Registered new tserver with Master: 2a86a28876eb407596a0452c86e6dd45 (127.6.212.3:41957)
I20260924 11:34:44.732936 14599 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.212.3@KRBTEST.COM'} at 127.6.212.3:39085
I20260924 11:34:44.745682 6992 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 11:34:44 dist-test-slave-cq7c krb5kdc[14555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249684, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 11:34:44 dist-test-slave-cq7c krb5kdc[14555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:34:44 dist-test-slave-cq7c krb5kdc[14555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 11:34:44.796319 14695 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:34:44.785479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43390 (local address 127.6.212.62:39467)
0924 11:34:44.785698 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:44.785706 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:44.785932 (+ 226us) server_negotiation.cc:407] Connection header received
0924 11:34:44.786142 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:44.786146 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:44.786209 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:44.786320 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:44.787691 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:44.788409 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:44.789753 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:44.790196 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:44.790334 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:44.795963 (+ 5629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43390: BlockingRecv error: recv got EOF from 127.0.0.1:43390 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 24 11:34:44 dist-test-slave-cq7c krb5kdc[14555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249684, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM
I20260924 11:34:44.812378 14695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:44.799230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43400 (local address 127.6.212.62:39467)
0924 11:34:44.799473 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:44.799481 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:44.799740 (+ 259us) server_negotiation.cc:407] Connection header received
0924 11:34:44.799931 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:44.799937 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:44.800022 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:44.800123 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:44.801255 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:44.802040 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:44.803452 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:44.803848 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:44.803984 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:44.807491 (+ 3507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:44.807496 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:44.807499 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:44.807531 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:44.809821 (+ 2290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:44.810482 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:44.810488 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:44.810492 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:44.810612 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:44.810992 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:44.810997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:44.811000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:44.811278 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:44.811584 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:44.811850 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 11:34:44.812080 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 11:34:44.816087 14599 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43400:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:34:44.823163 14599 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:34:44.845731 15105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:44.835337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.2:37311 (local address 127.0.0.1:56300)
0924 11:34:44.837258 (+ 1921us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:44.837304 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:44.837621 (+ 317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:44.838751 (+ 1130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:44.838758 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:44.838791 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:34:44.839208 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:44.839219 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:44.841453 (+ 2234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:44.841461 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:44.843608 (+ 2147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:44.843619 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:44.844823 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:44.844829 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:44.845011 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:44.845070 (+ 59us) client_negotiation.cc:769] Sending connection context
0924 11:34:44.845202 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 11:34:44.845394 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1563,"thread_start_us":1400,"threads_started":1}
I20260924 11:34:44.845723 15103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:44.835423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56300 (local address 127.6.212.2:37311)
0924 11:34:44.836061 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:44.836068 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:44.837420 (+ 1352us) server_negotiation.cc:407] Connection header received
0924 11:34:44.837890 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:44.837902 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:44.838250 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:44.838469 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:34:44.840125 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:44.841327 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:44.843750 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:44.844621 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:44.844776 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:44.845000 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:44.845251 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 11:34:44.845428 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":332,"threads_started":1}
I20260924 11:34:44.847795 15106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:44.835875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50376 (local address 127.6.212.3:41957)
0924 11:34:44.838740 (+ 2865us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:44.838752 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:44.838791 (+ 39us) server_negotiation.cc:407] Connection header received
0924 11:34:44.838904 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:44.838913 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:44.839172 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:44.839447 (+ 275us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:34:44.840487 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:44.842029 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:44.844797 (+ 2768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:44.846239 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:44.846361 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:44.846580 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:44.847303 (+ 723us) server_negotiation.cc:299] Negotiation successful
0924 11:34:44.847482 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2674,"thread_start_us":2497,"threads_started":1}
I20260924 11:34:44.848062 15104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:44.836897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:42819 (local address 127.0.0.1:52550)
0924 11:34:44.839806 (+ 2909us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:44.839828 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:44.839965 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:44.840761 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:44.840765 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:44.840789 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:34:44.841134 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:44.841144 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:44.842997 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:44.843008 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:44.845293 (+ 2285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:44.845307 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:44.847272 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:44.847276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:44.847459 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:44.847481 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 11:34:44.847576 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 11:34:44.847698 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2735,"mutex_wait_us":3,"thread_start_us":742,"threads_started":1}
I20260924 11:34:44.848070 15108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:44.837459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52550 (local address 127.6.212.1:42819)
0924 11:34:44.840218 (+ 2759us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:44.840228 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:44.840253 (+ 25us) server_negotiation.cc:407] Connection header received
0924 11:34:44.840347 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:44.840354 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:44.840555 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:44.840762 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:34:44.841304 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:44.842811 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:44.845466 (+ 2655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:44.846847 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:44.846982 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:44.847179 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:44.847637 (+ 458us) server_negotiation.cc:299] Negotiation successful
0924 11:34:44.847782 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2614,"thread_start_us":2487,"threads_started":1}
I20260924 11:34:44.848872 15107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:44.835429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.3:41957 (local address 127.0.0.1:50376)
0924 11:34:44.837939 (+ 2510us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:44.837960 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:44.838090 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:44.839699 (+ 1609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:44.839704 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:44.839729 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:34:44.840175 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:44.840186 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:44.842187 (+ 2001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:44.842197 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:44.844616 (+ 2419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:44.844630 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:44.846724 (+ 2094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:44.846732 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:44.846896 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:44.846943 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 11:34:44.848380 (+ 1437us) client_negotiation.cc:240] Negotiation successful
0924 11:34:44.848507 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2369,"mutex_wait_us":1,"thread_start_us":4074,"threads_started":1}
I20260924 11:34:44.849920 14876 tablet_service.cc:1467] Processing CreateTablet for tablet 9fdae3bf62164361897cb221083d67f9 (DEFAULT_TABLE table=test-table [id=b794fed2beed4a0099723cff56f795a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:34:44.851411 14876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fdae3bf62164361897cb221083d67f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:34:44.851228 14737 tablet_service.cc:1467] Processing CreateTablet for tablet 9fdae3bf62164361897cb221083d67f9 (DEFAULT_TABLE table=test-table [id=b794fed2beed4a0099723cff56f795a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:34:44.852487 14737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fdae3bf62164361897cb221083d67f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:34:44.852151 15015 tablet_service.cc:1467] Processing CreateTablet for tablet 9fdae3bf62164361897cb221083d67f9 (DEFAULT_TABLE table=test-table [id=b794fed2beed4a0099723cff56f795a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:34:44.853420 15015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fdae3bf62164361897cb221083d67f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:34:44.863346 15109 tablet_bootstrap.cc:492] T 9fdae3bf62164361897cb221083d67f9 P 4dddd225a8d649fabae7c12fa9fe99cc: Bootstrap starting.
I20260924 11:34:44.867244 15109 tablet_bootstrap.cc:654] T 9fdae3bf62164361897cb221083d67f9 P 4dddd225a8d649fabae7c12fa9fe99cc: Neither blocks nor log segments found. Creating new log.
I20260924 11:34:44.867522 15111 tablet_bootstrap.cc:492] T 9fdae3bf62164361897cb221083d67f9 P 2a86a28876eb407596a0452c86e6dd45: Bootstrap starting.
I20260924 11:34:44.868734 15109 log.cc:824] T 9fdae3bf62164361897cb221083d67f9 P 4dddd225a8d649fabae7c12fa9fe99cc: Log is configured to *not* fsync() on all Append() calls
I20260924 11:34:44.869217 15110 tablet_bootstrap.cc:492] T 9fdae3bf62164361897cb221083d67f9 P 432ac21283b94a269b47bdb8f90040b0: Bootstrap starting.
I20260924 11:34:44.871520 15111 tablet_bootstrap.cc:654] T 9fdae3bf62164361897cb221083d67f9 P 2a86a28876eb407596a0452c86e6dd45: Neither blocks nor log segments found. Creating new log.
I20260924 11:34:44.872032 15110 tablet_bootstrap.cc:654] T 9fdae3bf62164361897cb221083d67f9 P 432ac21283b94a269b47bdb8f90040b0: Neither blocks nor log segments found. Creating new log.
I20260924 11:34:44.872653 15111 log.cc:824] T 9fdae3bf62164361897cb221083d67f9 P 2a86a28876eb407596a0452c86e6dd45: Log is configured to *not* fsync() on all Append() calls
I20260924 11:34:44.873051 15110 log.cc:824] T 9fdae3bf62164361897cb221083d67f9 P 432ac21283b94a269b47bdb8f90040b0: Log is configured to *not* fsync() on all Append() calls
I20260924 11:34:44.874375 15109 tablet_bootstrap.cc:492] T 9fdae3bf62164361897cb221083d67f9 P 4dddd225a8d649fabae7c12fa9fe99cc: No bootstrap required, opened a new log
I20260924 11:34:44.874748 15109 ts_tablet_manager.cc:1397] T 9fdae3bf62164361897cb221083d67f9 P 4dddd225a8d649fabae7c12fa9fe99cc: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260924 11:34:44.877008 15111 tablet_bootstrap.cc:492] T 9fdae3bf62164361897cb221083d67f9 P 2a86a28876eb407596a0452c86e6dd45: No bootstrap required, opened a new log
I20260924 11:34:44.877008 15110 tablet_bootstrap.cc:492] T 9fdae3bf62164361897cb221083d67f9 P 432ac21283b94a269b47bdb8f90040b0: No bootstrap required, opened a new log
I20260924 11:34:44.877270 15111 ts_tablet_manager.cc:1397] T 9fdae3bf62164361897cb221083d67f9 P 2a86a28876eb407596a0452c86e6dd45: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260924 11:34:44.877270 15110 ts_tablet_manager.cc:1397] T 9fdae3bf62164361897cb221083d67f9 P 432ac21283b94a269b47bdb8f90040b0: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 11:34:44.882164 15109 raft_consensus.cc:348] T 9fdae3bf62164361897cb221083d67f9 P 4dddd225a8d649fabae7c12fa9fe99cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } } peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } }
I20260924 11:34:44.882529 15109 raft_consensus.cc:374] T 9fdae3bf62164361897cb221083d67f9 P 4dddd225a8d649fabae7c12fa9fe99cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:34:44.882659 15109 raft_consensus.cc:729] T 9fdae3bf62164361897cb221083d67f9 P 4dddd225a8d649fabae7c12fa9fe99cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dddd225a8d649fabae7c12fa9fe99cc, State: Initialized, Role: FOLLOWER
I20260924 11:34:44.883287 15109 consensus_queue.cc:260] T 9fdae3bf62164361897cb221083d67f9 P 4dddd225a8d649fabae7c12fa9fe99cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } } peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } }
I20260924 11:34:44.885108 15109 ts_tablet_manager.cc:1428] T 9fdae3bf62164361897cb221083d67f9 P 4dddd225a8d649fabae7c12fa9fe99cc: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260924 11:34:44.887688 15111 raft_consensus.cc:348] T 9fdae3bf62164361897cb221083d67f9 P 2a86a28876eb407596a0452c86e6dd45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } } peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } }
I20260924 11:34:44.888127 15111 raft_consensus.cc:374] T 9fdae3bf62164361897cb221083d67f9 P 2a86a28876eb407596a0452c86e6dd45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:34:44.888224 15111 raft_consensus.cc:729] T 9fdae3bf62164361897cb221083d67f9 P 2a86a28876eb407596a0452c86e6dd45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a86a28876eb407596a0452c86e6dd45, State: Initialized, Role: FOLLOWER
I20260924 11:34:44.888880 15111 consensus_queue.cc:260] T 9fdae3bf62164361897cb221083d67f9 P 2a86a28876eb407596a0452c86e6dd45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } } peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } }
I20260924 11:34:44.889133 15110 raft_consensus.cc:348] T 9fdae3bf62164361897cb221083d67f9 P 432ac21283b94a269b47bdb8f90040b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } } peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } }
I20260924 11:34:44.889623 15110 raft_consensus.cc:374] T 9fdae3bf62164361897cb221083d67f9 P 432ac21283b94a269b47bdb8f90040b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:34:44.889739 15110 raft_consensus.cc:729] T 9fdae3bf62164361897cb221083d67f9 P 432ac21283b94a269b47bdb8f90040b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 432ac21283b94a269b47bdb8f90040b0, State: Initialized, Role: FOLLOWER
I20260924 11:34:44.890044 15085 heartbeater.cc:503] Master 127.6.212.62:39467 was elected leader, sending a full tablet report...
I20260924 11:34:44.890311 15110 consensus_queue.cc:260] T 9fdae3bf62164361897cb221083d67f9 P 432ac21283b94a269b47bdb8f90040b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } } peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } }
I20260924 11:34:44.890723 15111 ts_tablet_manager.cc:1428] T 9fdae3bf62164361897cb221083d67f9 P 2a86a28876eb407596a0452c86e6dd45: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260924 11:34:44.891381 14943 heartbeater.cc:503] Master 127.6.212.62:39467 was elected leader, sending a full tablet report...
I20260924 11:34:44.891981 15110 ts_tablet_manager.cc:1428] T 9fdae3bf62164361897cb221083d67f9 P 432ac21283b94a269b47bdb8f90040b0: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260924 11:34:44.933830 15115 raft_consensus.cc:482] T 9fdae3bf62164361897cb221083d67f9 P 4dddd225a8d649fabae7c12fa9fe99cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:34:44.934190 15115 raft_consensus.cc:504] T 9fdae3bf62164361897cb221083d67f9 P 4dddd225a8d649fabae7c12fa9fe99cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } } peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } }
I20260924 11:34:44.936061 15115 leader_election.cc:288] T 9fdae3bf62164361897cb221083d67f9 P 4dddd225a8d649fabae7c12fa9fe99cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 432ac21283b94a269b47bdb8f90040b0 (127.6.212.2:37311), 2a86a28876eb407596a0452c86e6dd45 (127.6.212.3:41957)
W20260924 11:34:44.944100 14808 tablet.cc:2369] T 9fdae3bf62164361897cb221083d67f9 P 4dddd225a8d649fabae7c12fa9fe99cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:34:44.945006 15106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:44.936426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:45405 (local address 127.6.212.3:41957)
0924 11:34:44.936646 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:44.936655 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:44.937412 (+ 757us) server_negotiation.cc:407] Connection header received
0924 11:34:44.937636 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:44.937639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:44.937724 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:44.937819 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:34:44.938627 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:44.939753 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:44.942723 (+ 2970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:44.943980 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:44.944099 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:44.944209 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:44.944599 (+ 390us) server_negotiation.cc:299] Negotiation successful
0924 11:34:44.944748 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 11:34:44.945120 15103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:44.936505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:59973 (local address 127.6.212.2:37311)
0924 11:34:44.936723 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:44.936732 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:44.937543 (+ 811us) server_negotiation.cc:407] Connection header received
0924 11:34:44.937787 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:44.937795 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:44.937874 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:44.938000 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:34:44.938649 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:44.939758 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:44.942723 (+ 2965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:44.943961 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:44.944099 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:44.944209 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:44.944749 (+ 540us) server_negotiation.cc:299] Negotiation successful
0924 11:34:44.944876 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 11:34:44.945102 15119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:44.936342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.3:41957 (local address 127.6.212.1:45405)
0924 11:34:44.937294 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:44.937310 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:44.937478 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:44.937856 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:44.937860 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:44.937880 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:34:44.938448 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:44.938466 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:44.939929 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:44.939934 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:44.942569 (+ 2635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:44.942579 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:44.944218 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:44.944225 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:44.944409 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:44.944449 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 11:34:44.944557 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 11:34:44.944681 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"mutex_wait_us":142,"thread_start_us":704,"threads_started":1}
I20260924 11:34:44.945425 15118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:44.936387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.2:37311 (local address 127.6.212.1:59973)
0924 11:34:44.937449 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:44.937463 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:44.937599 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:44.938045 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:44.938048 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:44.938074 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:34:44.938509 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:44.938521 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:44.939930 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:44.939934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:44.942565 (+ 2631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:44.942579 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:44.944396 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:44.944402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:44.944573 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:44.944605 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 11:34:44.944954 (+ 349us) client_negotiation.cc:240] Negotiation successful
0924 11:34:44.945097 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":902,"mutex_wait_us":175,"thread_start_us":543,"threads_started":1}
I20260924 11:34:44.946317 15039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fdae3bf62164361897cb221083d67f9" candidate_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a86a28876eb407596a0452c86e6dd45" is_pre_election: true
I20260924 11:34:44.946507 14896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fdae3bf62164361897cb221083d67f9" candidate_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "432ac21283b94a269b47bdb8f90040b0" is_pre_election: true
I20260924 11:34:44.946826 15039 raft_consensus.cc:2454] T 9fdae3bf62164361897cb221083d67f9 P 2a86a28876eb407596a0452c86e6dd45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dddd225a8d649fabae7c12fa9fe99cc in term 0.
I20260924 11:34:44.947084 14896 raft_consensus.cc:2454] T 9fdae3bf62164361897cb221083d67f9 P 432ac21283b94a269b47bdb8f90040b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dddd225a8d649fabae7c12fa9fe99cc in term 0.
I20260924 11:34:44.947571 14689 leader_election.cc:302] T 9fdae3bf62164361897cb221083d67f9 P 4dddd225a8d649fabae7c12fa9fe99cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a86a28876eb407596a0452c86e6dd45, 4dddd225a8d649fabae7c12fa9fe99cc; no voters:
I20260924 11:34:44.947988 15115 raft_consensus.cc:2781] T 9fdae3bf62164361897cb221083d67f9 P 4dddd225a8d649fabae7c12fa9fe99cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:34:44.948117 15115 raft_consensus.cc:482] T 9fdae3bf62164361897cb221083d67f9 P 4dddd225a8d649fabae7c12fa9fe99cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:34:44.948213 15115 raft_consensus.cc:3037] T 9fdae3bf62164361897cb221083d67f9 P 4dddd225a8d649fabae7c12fa9fe99cc [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:34:44.950366 15115 raft_consensus.cc:504] T 9fdae3bf62164361897cb221083d67f9 P 4dddd225a8d649fabae7c12fa9fe99cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } } peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } }
I20260924 11:34:44.950990 15115 leader_election.cc:288] T 9fdae3bf62164361897cb221083d67f9 P 4dddd225a8d649fabae7c12fa9fe99cc [CANDIDATE]: Term 1 election: Requested vote from peers 432ac21283b94a269b47bdb8f90040b0 (127.6.212.2:37311), 2a86a28876eb407596a0452c86e6dd45 (127.6.212.3:41957)
I20260924 11:34:44.951334 14896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fdae3bf62164361897cb221083d67f9" candidate_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "432ac21283b94a269b47bdb8f90040b0"
I20260924 11:34:44.951485 15039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fdae3bf62164361897cb221083d67f9" candidate_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a86a28876eb407596a0452c86e6dd45"
I20260924 11:34:44.951558 14896 raft_consensus.cc:3037] T 9fdae3bf62164361897cb221083d67f9 P 432ac21283b94a269b47bdb8f90040b0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:34:44.951711 15039 raft_consensus.cc:3037] T 9fdae3bf62164361897cb221083d67f9 P 2a86a28876eb407596a0452c86e6dd45 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:34:44.953714 14896 raft_consensus.cc:2454] T 9fdae3bf62164361897cb221083d67f9 P 432ac21283b94a269b47bdb8f90040b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dddd225a8d649fabae7c12fa9fe99cc in term 1.
I20260924 11:34:44.953900 15039 raft_consensus.cc:2454] T 9fdae3bf62164361897cb221083d67f9 P 2a86a28876eb407596a0452c86e6dd45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dddd225a8d649fabae7c12fa9fe99cc in term 1.
I20260924 11:34:44.954094 14690 leader_election.cc:302] T 9fdae3bf62164361897cb221083d67f9 P 4dddd225a8d649fabae7c12fa9fe99cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 432ac21283b94a269b47bdb8f90040b0, 4dddd225a8d649fabae7c12fa9fe99cc; no voters:
I20260924 11:34:44.954420 15115 raft_consensus.cc:2781] T 9fdae3bf62164361897cb221083d67f9 P 4dddd225a8d649fabae7c12fa9fe99cc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:34:44.954873 15115 raft_consensus.cc:686] T 9fdae3bf62164361897cb221083d67f9 P 4dddd225a8d649fabae7c12fa9fe99cc [term 1 LEADER]: Becoming Leader. State: Replica: 4dddd225a8d649fabae7c12fa9fe99cc, State: Running, Role: LEADER
I20260924 11:34:44.955427 15115 consensus_queue.cc:237] T 9fdae3bf62164361897cb221083d67f9 P 4dddd225a8d649fabae7c12fa9fe99cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } } peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } }
W20260924 11:34:44.958091 14945 tablet.cc:2369] T 9fdae3bf62164361897cb221083d67f9 P 432ac21283b94a269b47bdb8f90040b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:34:44.960180 14599 catalog_manager.cc:5495] T 9fdae3bf62164361897cb221083d67f9 P 4dddd225a8d649fabae7c12fa9fe99cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 4dddd225a8d649fabae7c12fa9fe99cc (127.6.212.1). New cstate: current_term: 1 leader_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } health_report { overall_health: HEALTHY } } }
W20260924 11:34:44.978411 15086 tablet.cc:2369] T 9fdae3bf62164361897cb221083d67f9 P 2a86a28876eb407596a0452c86e6dd45: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:34:44.998867 15108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:44.993502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52554 (local address 127.6.212.1:42819)
0924 11:34:44.993690 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:44.993694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:44.993712 (+ 18us) server_negotiation.cc:407] Connection header received
0924 11:34:44.993889 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:44.993893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:44.993986 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:44.994107 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:34:44.994448 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:44.995149 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:44.996595 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:44.997061 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:44.997153 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:44.997620 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:34:44.998249 (+ 629us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:34:44.998376 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:44.998442 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 11:34:44.998502 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 11:34:45.007467 15039 raft_consensus.cc:1261] T 9fdae3bf62164361897cb221083d67f9 P 2a86a28876eb407596a0452c86e6dd45 [term 1 FOLLOWER]: Refusing update from remote peer 4dddd225a8d649fabae7c12fa9fe99cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:34:45.007517 14896 raft_consensus.cc:1261] T 9fdae3bf62164361897cb221083d67f9 P 432ac21283b94a269b47bdb8f90040b0 [term 1 FOLLOWER]: Refusing update from remote peer 4dddd225a8d649fabae7c12fa9fe99cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:34:45.008440 15120 consensus_queue.cc:1035] T 9fdae3bf62164361897cb221083d67f9 P 4dddd225a8d649fabae7c12fa9fe99cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:34:45.008782 15115 consensus_queue.cc:1035] T 9fdae3bf62164361897cb221083d67f9 P 4dddd225a8d649fabae7c12fa9fe99cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:34:45.017946 15123 mvcc.cc:205] Tried to move back new op lower bound from 7332862709780512768 to 7332862709581484032. Current Snapshot: MvccSnapshot[applied={T|T < 7332862709780512768}]
I20260924 11:34:45.021639 15125 mvcc.cc:205] Tried to move back new op lower bound from 7332862709780512768 to 7332862709581484032. Current Snapshot: MvccSnapshot[applied={T|T < 7332862709780512768}]
I20260924 11:34:45.036660 14600 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:34:45.036906 14600 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:34:45.040480 14600 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c9bf26035dc4b5e9edde52527a113e7: Sending DeleteTablet for 3 replicas of tablet 9fdae3bf62164361897cb221083d67f9
I20260924 11:34:45.041544 14876 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fdae3bf62164361897cb221083d67f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:34:45 UTC) from {username='oryx', principal='oryx/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:56300
I20260924 11:34:45.041546 15015 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fdae3bf62164361897cb221083d67f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:34:45 UTC) from {username='oryx', principal='oryx/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:50376
I20260924 11:34:45.042857 14737 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fdae3bf62164361897cb221083d67f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:34:45 UTC) from {username='oryx', principal='oryx/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:52550
I20260924 11:34:45.051860 15134 tablet_replica.cc:330] T 9fdae3bf62164361897cb221083d67f9 P 4dddd225a8d649fabae7c12fa9fe99cc: stopping tablet replica
I20260924 11:34:45.051602 14599 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43400:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:34:45.052491 14599 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:34:45.052531 15134 raft_consensus.cc:2229] T 9fdae3bf62164361897cb221083d67f9 P 4dddd225a8d649fabae7c12fa9fe99cc [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:34:45.053313 15134 raft_consensus.cc:2258] T 9fdae3bf62164361897cb221083d67f9 P 4dddd225a8d649fabae7c12fa9fe99cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:34:45.062469 14875 tablet_service.cc:1467] Processing CreateTablet for tablet e6e9ef978ee048979379e976c422f03b (DEFAULT_TABLE table=test-table [id=1d3fc936fac14f2aa1e3cec05baafd97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:34:45.062800 15134 ts_tablet_manager.cc:1905] T 9fdae3bf62164361897cb221083d67f9 P 4dddd225a8d649fabae7c12fa9fe99cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:34:45.063335 14875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6e9ef978ee048979379e976c422f03b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:34:45.068087 15014 tablet_service.cc:1467] Processing CreateTablet for tablet e6e9ef978ee048979379e976c422f03b (DEFAULT_TABLE table=test-table [id=1d3fc936fac14f2aa1e3cec05baafd97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:34:45.068836 15014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6e9ef978ee048979379e976c422f03b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:34:45.070389 15134 ts_tablet_manager.cc:1918] T 9fdae3bf62164361897cb221083d67f9 P 4dddd225a8d649fabae7c12fa9fe99cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:34:45.070542 15134 log.cc:1196] T 9fdae3bf62164361897cb221083d67f9 P 4dddd225a8d649fabae7c12fa9fe99cc: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-0/wal/wals/9fdae3bf62164361897cb221083d67f9
I20260924 11:34:45.071095 15134 ts_tablet_manager.cc:1939] T 9fdae3bf62164361897cb221083d67f9 P 4dddd225a8d649fabae7c12fa9fe99cc: Deleting consensus metadata
I20260924 11:34:45.072961 14584 catalog_manager.cc:4841] TS 4dddd225a8d649fabae7c12fa9fe99cc (127.6.212.1:42819): tablet 9fdae3bf62164361897cb221083d67f9 (table test-table [id=b794fed2beed4a0099723cff56f795a7]) successfully deleted
I20260924 11:34:45.072837 14737 tablet_service.cc:1467] Processing CreateTablet for tablet e6e9ef978ee048979379e976c422f03b (DEFAULT_TABLE table=test-table [id=1d3fc936fac14f2aa1e3cec05baafd97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:34:45.073479 14737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6e9ef978ee048979379e976c422f03b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:34:45.093173 15111 tablet_bootstrap.cc:492] T e6e9ef978ee048979379e976c422f03b P 2a86a28876eb407596a0452c86e6dd45: Bootstrap starting.
I20260924 11:34:45.095304 15111 tablet_bootstrap.cc:654] T e6e9ef978ee048979379e976c422f03b P 2a86a28876eb407596a0452c86e6dd45: Neither blocks nor log segments found. Creating new log.
I20260924 11:34:45.095346 15110 tablet_bootstrap.cc:492] T e6e9ef978ee048979379e976c422f03b P 432ac21283b94a269b47bdb8f90040b0: Bootstrap starting.
I20260924 11:34:45.097157 15109 tablet_bootstrap.cc:492] T e6e9ef978ee048979379e976c422f03b P 4dddd225a8d649fabae7c12fa9fe99cc: Bootstrap starting.
I20260924 11:34:45.097630 15110 tablet_bootstrap.cc:654] T e6e9ef978ee048979379e976c422f03b P 432ac21283b94a269b47bdb8f90040b0: Neither blocks nor log segments found. Creating new log.
I20260924 11:34:45.097692 15132 tablet_replica.cc:330] T 9fdae3bf62164361897cb221083d67f9 P 432ac21283b94a269b47bdb8f90040b0: stopping tablet replica
I20260924 11:34:45.098448 15133 tablet_replica.cc:330] T 9fdae3bf62164361897cb221083d67f9 P 2a86a28876eb407596a0452c86e6dd45: stopping tablet replica
I20260924 11:34:45.098862 15109 tablet_bootstrap.cc:654] T e6e9ef978ee048979379e976c422f03b P 4dddd225a8d649fabae7c12fa9fe99cc: Neither blocks nor log segments found. Creating new log.
I20260924 11:34:45.098888 15133 raft_consensus.cc:2229] T 9fdae3bf62164361897cb221083d67f9 P 2a86a28876eb407596a0452c86e6dd45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:34:45.099463 15133 raft_consensus.cc:2258] T 9fdae3bf62164361897cb221083d67f9 P 2a86a28876eb407596a0452c86e6dd45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:34:45.099998 15132 raft_consensus.cc:2229] T 9fdae3bf62164361897cb221083d67f9 P 432ac21283b94a269b47bdb8f90040b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:34:45.100576 15132 raft_consensus.cc:2258] T 9fdae3bf62164361897cb221083d67f9 P 432ac21283b94a269b47bdb8f90040b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:34:45.105269 15133 ts_tablet_manager.cc:1905] T 9fdae3bf62164361897cb221083d67f9 P 2a86a28876eb407596a0452c86e6dd45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:34:45.108440 15132 ts_tablet_manager.cc:1905] T 9fdae3bf62164361897cb221083d67f9 P 432ac21283b94a269b47bdb8f90040b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:34:45.108762 15110 tablet_bootstrap.cc:492] T e6e9ef978ee048979379e976c422f03b P 432ac21283b94a269b47bdb8f90040b0: No bootstrap required, opened a new log
I20260924 11:34:45.109186 15110 ts_tablet_manager.cc:1397] T e6e9ef978ee048979379e976c422f03b P 432ac21283b94a269b47bdb8f90040b0: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.004s
I20260924 11:34:45.110044 15110 raft_consensus.cc:348] T e6e9ef978ee048979379e976c422f03b P 432ac21283b94a269b47bdb8f90040b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } } peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } }
I20260924 11:34:45.110394 15110 raft_consensus.cc:374] T e6e9ef978ee048979379e976c422f03b P 432ac21283b94a269b47bdb8f90040b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:34:45.110535 15110 raft_consensus.cc:729] T e6e9ef978ee048979379e976c422f03b P 432ac21283b94a269b47bdb8f90040b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 432ac21283b94a269b47bdb8f90040b0, State: Initialized, Role: FOLLOWER
I20260924 11:34:45.110868 15110 consensus_queue.cc:260] T e6e9ef978ee048979379e976c422f03b P 432ac21283b94a269b47bdb8f90040b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } } peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } }
I20260924 11:34:45.114702 15110 ts_tablet_manager.cc:1428] T e6e9ef978ee048979379e976c422f03b P 432ac21283b94a269b47bdb8f90040b0: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20260924 11:34:45.118088 15109 tablet_bootstrap.cc:492] T e6e9ef978ee048979379e976c422f03b P 4dddd225a8d649fabae7c12fa9fe99cc: No bootstrap required, opened a new log
I20260924 11:34:45.118300 15109 ts_tablet_manager.cc:1397] T e6e9ef978ee048979379e976c422f03b P 4dddd225a8d649fabae7c12fa9fe99cc: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.006s
I20260924 11:34:45.118866 15133 ts_tablet_manager.cc:1918] T 9fdae3bf62164361897cb221083d67f9 P 2a86a28876eb407596a0452c86e6dd45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:34:45.119170 15109 raft_consensus.cc:348] T e6e9ef978ee048979379e976c422f03b P 4dddd225a8d649fabae7c12fa9fe99cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } } peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } }
I20260924 11:34:45.119488 15109 raft_consensus.cc:374] T e6e9ef978ee048979379e976c422f03b P 4dddd225a8d649fabae7c12fa9fe99cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:34:45.119634 15109 raft_consensus.cc:729] T e6e9ef978ee048979379e976c422f03b P 4dddd225a8d649fabae7c12fa9fe99cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dddd225a8d649fabae7c12fa9fe99cc, State: Initialized, Role: FOLLOWER
I20260924 11:34:45.119742 15133 log.cc:1196] T 9fdae3bf62164361897cb221083d67f9 P 2a86a28876eb407596a0452c86e6dd45: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-2/wal/wals/9fdae3bf62164361897cb221083d67f9
I20260924 11:34:45.119989 15109 consensus_queue.cc:260] T e6e9ef978ee048979379e976c422f03b P 4dddd225a8d649fabae7c12fa9fe99cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } } peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } }
I20260924 11:34:45.120396 15133 ts_tablet_manager.cc:1939] T 9fdae3bf62164361897cb221083d67f9 P 2a86a28876eb407596a0452c86e6dd45: Deleting consensus metadata
I20260924 11:34:45.122476 14586 catalog_manager.cc:4841] TS 2a86a28876eb407596a0452c86e6dd45 (127.6.212.3:41957): tablet 9fdae3bf62164361897cb221083d67f9 (table test-table [id=b794fed2beed4a0099723cff56f795a7]) successfully deleted
I20260924 11:34:45.123546 15109 ts_tablet_manager.cc:1428] T e6e9ef978ee048979379e976c422f03b P 4dddd225a8d649fabae7c12fa9fe99cc: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260924 11:34:45.124068 15111 tablet_bootstrap.cc:492] T e6e9ef978ee048979379e976c422f03b P 2a86a28876eb407596a0452c86e6dd45: No bootstrap required, opened a new log
I20260924 11:34:45.124236 15111 ts_tablet_manager.cc:1397] T e6e9ef978ee048979379e976c422f03b P 2a86a28876eb407596a0452c86e6dd45: Time spent bootstrapping tablet: real 0.031s user 0.014s sys 0.013s
I20260924 11:34:45.125039 15111 raft_consensus.cc:348] T e6e9ef978ee048979379e976c422f03b P 2a86a28876eb407596a0452c86e6dd45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } } peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } }
I20260924 11:34:45.125290 15111 raft_consensus.cc:374] T e6e9ef978ee048979379e976c422f03b P 2a86a28876eb407596a0452c86e6dd45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:34:45.125398 15111 raft_consensus.cc:729] T e6e9ef978ee048979379e976c422f03b P 2a86a28876eb407596a0452c86e6dd45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a86a28876eb407596a0452c86e6dd45, State: Initialized, Role: FOLLOWER
I20260924 11:34:45.125665 15111 consensus_queue.cc:260] T e6e9ef978ee048979379e976c422f03b P 2a86a28876eb407596a0452c86e6dd45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } } peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } }
I20260924 11:34:45.126135 15132 ts_tablet_manager.cc:1918] T 9fdae3bf62164361897cb221083d67f9 P 432ac21283b94a269b47bdb8f90040b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:34:45.126307 15132 log.cc:1196] T 9fdae3bf62164361897cb221083d67f9 P 432ac21283b94a269b47bdb8f90040b0: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-1/wal/wals/9fdae3bf62164361897cb221083d67f9
I20260924 11:34:45.126905 15132 ts_tablet_manager.cc:1939] T 9fdae3bf62164361897cb221083d67f9 P 432ac21283b94a269b47bdb8f90040b0: Deleting consensus metadata
I20260924 11:34:45.127010 15111 ts_tablet_manager.cc:1428] T e6e9ef978ee048979379e976c422f03b P 2a86a28876eb407596a0452c86e6dd45: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260924 11:34:45.128728 14587 catalog_manager.cc:4841] TS 432ac21283b94a269b47bdb8f90040b0 (127.6.212.2:37311): tablet 9fdae3bf62164361897cb221083d67f9 (table test-table [id=b794fed2beed4a0099723cff56f795a7]) successfully deleted
I20260924 11:34:45.135957 15116 raft_consensus.cc:482] T e6e9ef978ee048979379e976c422f03b P 2a86a28876eb407596a0452c86e6dd45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:34:45.136170 15116 raft_consensus.cc:504] T e6e9ef978ee048979379e976c422f03b P 2a86a28876eb407596a0452c86e6dd45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } } peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } }
I20260924 11:34:45.137791 15116 leader_election.cc:288] T e6e9ef978ee048979379e976c422f03b P 2a86a28876eb407596a0452c86e6dd45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4dddd225a8d649fabae7c12fa9fe99cc (127.6.212.1:42819), 432ac21283b94a269b47bdb8f90040b0 (127.6.212.2:37311)
I20260924 11:34:45.145475 14974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:45.138156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:42819 (local address 127.6.212.3:48881)
0924 11:34:45.138637 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:45.138662 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:45.138809 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:45.139828 (+ 1019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:45.139834 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:45.139867 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:34:45.140074 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:45.140088 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:45.141385 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:45.141390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:45.143014 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:45.143028 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:45.144661 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:45.144667 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:45.144869 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:45.144904 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 11:34:45.145018 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 11:34:45.145154 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":24}
I20260924 11:34:45.145591 15108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:45.139254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:48881 (local address 127.6.212.1:42819)
0924 11:34:45.139498 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:45.139504 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:45.139524 (+ 20us) server_negotiation.cc:407] Connection header received
0924 11:34:45.139593 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:45.139598 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:45.139684 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:45.139791 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:34:45.140196 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:45.141258 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:45.143174 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:45.144481 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:45.144634 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:45.144745 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:45.145094 (+ 349us) server_negotiation.cc:299] Negotiation successful
0924 11:34:45.145231 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 11:34:45.146227 15103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:45.138318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:57969 (local address 127.6.212.2:37311)
0924 11:34:45.138491 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:45.138498 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:45.139917 (+ 1419us) server_negotiation.cc:407] Connection header received
0924 11:34:45.140165 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:45.140172 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:45.140297 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:45.140394 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:34:45.140990 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:45.142100 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:45.144096 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:45.145085 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:45.145190 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:45.145287 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:45.145822 (+ 535us) server_negotiation.cc:299] Negotiation successful
0924 11:34:45.145972 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 11:34:45.146947 15138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:45.138227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.2:37311 (local address 127.6.212.3:57969)
0924 11:34:45.139825 (+ 1598us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:45.139841 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:45.140009 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:45.140441 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:45.140448 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:45.140475 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:34:45.140819 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:45.140830 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:45.142240 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:45.142246 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:45.143948 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:45.143959 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:45.145436 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:45.145441 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:45.145623 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:45.145662 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 11:34:45.146388 (+ 726us) client_negotiation.cc:240] Negotiation successful
0924 11:34:45.146537 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1456,"mutex_wait_us":3,"thread_start_us":1324,"threads_started":1}
I20260924 11:34:45.146693 14757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6e9ef978ee048979379e976c422f03b" candidate_uuid: "2a86a28876eb407596a0452c86e6dd45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" is_pre_election: true
I20260924 11:34:45.147192 14757 raft_consensus.cc:2454] T e6e9ef978ee048979379e976c422f03b P 4dddd225a8d649fabae7c12fa9fe99cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a86a28876eb407596a0452c86e6dd45 in term 0.
I20260924 11:34:45.147565 14896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6e9ef978ee048979379e976c422f03b" candidate_uuid: "2a86a28876eb407596a0452c86e6dd45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "432ac21283b94a269b47bdb8f90040b0" is_pre_election: true
I20260924 11:34:45.147825 14896 raft_consensus.cc:2454] T e6e9ef978ee048979379e976c422f03b P 432ac21283b94a269b47bdb8f90040b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a86a28876eb407596a0452c86e6dd45 in term 0.
I20260924 11:34:45.148176 14966 leader_election.cc:302] T e6e9ef978ee048979379e976c422f03b P 2a86a28876eb407596a0452c86e6dd45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a86a28876eb407596a0452c86e6dd45, 4dddd225a8d649fabae7c12fa9fe99cc; no voters:
I20260924 11:34:45.148612 15116 raft_consensus.cc:2781] T e6e9ef978ee048979379e976c422f03b P 2a86a28876eb407596a0452c86e6dd45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:34:45.148798 15116 raft_consensus.cc:482] T e6e9ef978ee048979379e976c422f03b P 2a86a28876eb407596a0452c86e6dd45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:34:45.148893 15116 raft_consensus.cc:3037] T e6e9ef978ee048979379e976c422f03b P 2a86a28876eb407596a0452c86e6dd45 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:34:45.152012 15116 raft_consensus.cc:504] T e6e9ef978ee048979379e976c422f03b P 2a86a28876eb407596a0452c86e6dd45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } } peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } }
I20260924 11:34:45.152717 15116 leader_election.cc:288] T e6e9ef978ee048979379e976c422f03b P 2a86a28876eb407596a0452c86e6dd45 [CANDIDATE]: Term 1 election: Requested vote from peers 4dddd225a8d649fabae7c12fa9fe99cc (127.6.212.1:42819), 432ac21283b94a269b47bdb8f90040b0 (127.6.212.2:37311)
I20260924 11:34:45.153180 14757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6e9ef978ee048979379e976c422f03b" candidate_uuid: "2a86a28876eb407596a0452c86e6dd45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dddd225a8d649fabae7c12fa9fe99cc"
I20260924 11:34:45.153219 14896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6e9ef978ee048979379e976c422f03b" candidate_uuid: "2a86a28876eb407596a0452c86e6dd45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "432ac21283b94a269b47bdb8f90040b0"
I20260924 11:34:45.153398 14757 raft_consensus.cc:3037] T e6e9ef978ee048979379e976c422f03b P 4dddd225a8d649fabae7c12fa9fe99cc [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:34:45.153429 14896 raft_consensus.cc:3037] T e6e9ef978ee048979379e976c422f03b P 432ac21283b94a269b47bdb8f90040b0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:34:45.155879 14757 raft_consensus.cc:2454] T e6e9ef978ee048979379e976c422f03b P 4dddd225a8d649fabae7c12fa9fe99cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a86a28876eb407596a0452c86e6dd45 in term 1.
I20260924 11:34:45.156325 14966 leader_election.cc:302] T e6e9ef978ee048979379e976c422f03b P 2a86a28876eb407596a0452c86e6dd45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a86a28876eb407596a0452c86e6dd45, 4dddd225a8d649fabae7c12fa9fe99cc; no voters:
I20260924 11:34:45.156402 14896 raft_consensus.cc:2454] T e6e9ef978ee048979379e976c422f03b P 432ac21283b94a269b47bdb8f90040b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a86a28876eb407596a0452c86e6dd45 in term 1.
I20260924 11:34:45.156728 15116 raft_consensus.cc:2781] T e6e9ef978ee048979379e976c422f03b P 2a86a28876eb407596a0452c86e6dd45 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:34:45.157191 15116 raft_consensus.cc:686] T e6e9ef978ee048979379e976c422f03b P 2a86a28876eb407596a0452c86e6dd45 [term 1 LEADER]: Becoming Leader. State: Replica: 2a86a28876eb407596a0452c86e6dd45, State: Running, Role: LEADER
I20260924 11:34:45.157599 15116 consensus_queue.cc:237] T e6e9ef978ee048979379e976c422f03b P 2a86a28876eb407596a0452c86e6dd45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } } peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } }
I20260924 11:34:45.160971 14600 catalog_manager.cc:5495] T e6e9ef978ee048979379e976c422f03b P 2a86a28876eb407596a0452c86e6dd45 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a86a28876eb407596a0452c86e6dd45 (127.6.212.3). New cstate: current_term: 1 leader_uuid: "2a86a28876eb407596a0452c86e6dd45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } health_report { overall_health: HEALTHY } } }
Sep 24 11:34:45 dist-test-slave-cq7c krb5kdc[14555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249682, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.212.62@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM
I20260924 11:34:45.227224 14695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:45.213721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43406 (local address 127.6.212.62:39467)
0924 11:34:45.213968 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:45.213976 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:45.214538 (+ 562us) server_negotiation.cc:407] Connection header received
0924 11:34:45.214740 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:45.214751 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:45.214802 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:45.214899 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:45.216067 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:45.216808 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:45.218014 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:45.218554 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:45.218679 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:45.221887 (+ 3208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:45.221891 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:45.221894 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:45.221920 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:45.224459 (+ 2539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:45.225175 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:45.225180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:45.225183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:45.225330 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:45.225667 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:45.225674 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:45.225678 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:45.225913 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:45.226084 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:45.226469 (+ 385us) server_negotiation.cc:299] Negotiation successful
0924 11:34:45.226835 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":170,"server-negotiator.queue_time_us":113}
I20260924 11:34:45.227247 15146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:45.213727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:39467 (local address 127.0.0.1:43406)
0924 11:34:45.214417 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:45.214453 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:45.214617 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:45.214923 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:45.214927 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:45.215679 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:45.215931 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:45.215941 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:45.216937 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:45.216943 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:45.217878 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:45.217889 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:45.218723 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:45.218729 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:45.218882 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:45.219560 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:45.219577 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:45.221706 (+ 2129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:45.224596 (+ 2890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:45.224608 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:45.224616 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:45.225011 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:45.225423 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:45.225436 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:45.225439 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:45.225565 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:45.226057 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:45.226065 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:45.226226 (+ 161us) client_negotiation.cc:769] Sending connection context
0924 11:34:45.226387 (+ 161us) client_negotiation.cc:240] Negotiation successful
0924 11:34:45.226724 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"mutex_wait_us":5,"thread_start_us":416,"threads_started":1}
I20260924 11:34:45.232964 14600 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:43406:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 11:34:45.234014 14600 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:34:45.240856 14737 tablet_service.cc:1467] Processing CreateTablet for tablet af5b73e6e44b4e6f9c610450c3f618d6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=adcccccd58da4caabc5fa8125ff36561]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 11:34:45.241086 15015 tablet_service.cc:1467] Processing CreateTablet for tablet af5b73e6e44b4e6f9c610450c3f618d6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=adcccccd58da4caabc5fa8125ff36561]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 11:34:45.241483 14737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af5b73e6e44b4e6f9c610450c3f618d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:34:45.241679 15015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af5b73e6e44b4e6f9c610450c3f618d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:34:45.244378 14875 tablet_service.cc:1467] Processing CreateTablet for tablet af5b73e6e44b4e6f9c610450c3f618d6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=adcccccd58da4caabc5fa8125ff36561]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 11:34:45.244892 14875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af5b73e6e44b4e6f9c610450c3f618d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:34:45.248952 15109 tablet_bootstrap.cc:492] T af5b73e6e44b4e6f9c610450c3f618d6 P 4dddd225a8d649fabae7c12fa9fe99cc: Bootstrap starting.
I20260924 11:34:45.250557 15110 tablet_bootstrap.cc:492] T af5b73e6e44b4e6f9c610450c3f618d6 P 432ac21283b94a269b47bdb8f90040b0: Bootstrap starting.
I20260924 11:34:45.250703 15109 tablet_bootstrap.cc:654] T af5b73e6e44b4e6f9c610450c3f618d6 P 4dddd225a8d649fabae7c12fa9fe99cc: Neither blocks nor log segments found. Creating new log.
I20260924 11:34:45.252151 15110 tablet_bootstrap.cc:654] T af5b73e6e44b4e6f9c610450c3f618d6 P 432ac21283b94a269b47bdb8f90040b0: Neither blocks nor log segments found. Creating new log.
I20260924 11:34:45.256624 15111 tablet_bootstrap.cc:492] T af5b73e6e44b4e6f9c610450c3f618d6 P 2a86a28876eb407596a0452c86e6dd45: Bootstrap starting.
I20260924 11:34:45.258520 15111 tablet_bootstrap.cc:654] T af5b73e6e44b4e6f9c610450c3f618d6 P 2a86a28876eb407596a0452c86e6dd45: Neither blocks nor log segments found. Creating new log.
I20260924 11:34:45.262771 15111 tablet_bootstrap.cc:492] T af5b73e6e44b4e6f9c610450c3f618d6 P 2a86a28876eb407596a0452c86e6dd45: No bootstrap required, opened a new log
I20260924 11:34:45.262929 15111 ts_tablet_manager.cc:1397] T af5b73e6e44b4e6f9c610450c3f618d6 P 2a86a28876eb407596a0452c86e6dd45: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 11:34:45.263437 15109 tablet_bootstrap.cc:492] T af5b73e6e44b4e6f9c610450c3f618d6 P 4dddd225a8d649fabae7c12fa9fe99cc: No bootstrap required, opened a new log
I20260924 11:34:45.263543 15110 tablet_bootstrap.cc:492] T af5b73e6e44b4e6f9c610450c3f618d6 P 432ac21283b94a269b47bdb8f90040b0: No bootstrap required, opened a new log
I20260924 11:34:45.263640 15109 ts_tablet_manager.cc:1397] T af5b73e6e44b4e6f9c610450c3f618d6 P 4dddd225a8d649fabae7c12fa9fe99cc: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20260924 11:34:45.263720 15110 ts_tablet_manager.cc:1397] T af5b73e6e44b4e6f9c610450c3f618d6 P 432ac21283b94a269b47bdb8f90040b0: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.008s
I20260924 11:34:45.263760 15111 raft_consensus.cc:348] T af5b73e6e44b4e6f9c610450c3f618d6 P 2a86a28876eb407596a0452c86e6dd45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } } peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } }
I20260924 11:34:45.264019 15111 raft_consensus.cc:374] T af5b73e6e44b4e6f9c610450c3f618d6 P 2a86a28876eb407596a0452c86e6dd45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:34:45.264088 15111 raft_consensus.cc:729] T af5b73e6e44b4e6f9c610450c3f618d6 P 2a86a28876eb407596a0452c86e6dd45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a86a28876eb407596a0452c86e6dd45, State: Initialized, Role: FOLLOWER
I20260924 11:34:45.264304 15111 consensus_queue.cc:260] T af5b73e6e44b4e6f9c610450c3f618d6 P 2a86a28876eb407596a0452c86e6dd45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } } peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } }
I20260924 11:34:45.264384 15109 raft_consensus.cc:348] T af5b73e6e44b4e6f9c610450c3f618d6 P 4dddd225a8d649fabae7c12fa9fe99cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } } peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } }
I20260924 11:34:45.264503 15110 raft_consensus.cc:348] T af5b73e6e44b4e6f9c610450c3f618d6 P 432ac21283b94a269b47bdb8f90040b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } } peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } }
I20260924 11:34:45.264635 15109 raft_consensus.cc:374] T af5b73e6e44b4e6f9c610450c3f618d6 P 4dddd225a8d649fabae7c12fa9fe99cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:34:45.264708 15109 raft_consensus.cc:729] T af5b73e6e44b4e6f9c610450c3f618d6 P 4dddd225a8d649fabae7c12fa9fe99cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dddd225a8d649fabae7c12fa9fe99cc, State: Initialized, Role: FOLLOWER
I20260924 11:34:45.264714 15110 raft_consensus.cc:374] T af5b73e6e44b4e6f9c610450c3f618d6 P 432ac21283b94a269b47bdb8f90040b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:34:45.264781 15110 raft_consensus.cc:729] T af5b73e6e44b4e6f9c610450c3f618d6 P 432ac21283b94a269b47bdb8f90040b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 432ac21283b94a269b47bdb8f90040b0, State: Initialized, Role: FOLLOWER
I20260924 11:34:45.264845 15111 ts_tablet_manager.cc:1428] T af5b73e6e44b4e6f9c610450c3f618d6 P 2a86a28876eb407596a0452c86e6dd45: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 11:34:45.264981 15109 consensus_queue.cc:260] T af5b73e6e44b4e6f9c610450c3f618d6 P 4dddd225a8d649fabae7c12fa9fe99cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } } peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } }
I20260924 11:34:45.265082 15110 consensus_queue.cc:260] T af5b73e6e44b4e6f9c610450c3f618d6 P 432ac21283b94a269b47bdb8f90040b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } } peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } }
I20260924 11:34:45.265162 15116 tablet_replica.cc:439] T af5b73e6e44b4e6f9c610450c3f618d6 P 2a86a28876eb407596a0452c86e6dd45: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } } peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } } }
I20260924 11:34:45.265448 15116 tablet_replica.cc:442] T af5b73e6e44b4e6f9c610450c3f618d6 P 2a86a28876eb407596a0452c86e6dd45: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:34:45.265398 15115 tablet_replica.cc:439] T af5b73e6e44b4e6f9c610450c3f618d6 P 4dddd225a8d649fabae7c12fa9fe99cc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } } peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } } }
I20260924 11:34:45.265645 15115 tablet_replica.cc:442] T af5b73e6e44b4e6f9c610450c3f618d6 P 4dddd225a8d649fabae7c12fa9fe99cc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:34:45.265506 15117 tablet_replica.cc:439] T af5b73e6e44b4e6f9c610450c3f618d6 P 432ac21283b94a269b47bdb8f90040b0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } } peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } } }
I20260924 11:34:45.265834 15117 tablet_replica.cc:442] T af5b73e6e44b4e6f9c610450c3f618d6 P 432ac21283b94a269b47bdb8f90040b0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:34:45.267158 15110 ts_tablet_manager.cc:1428] T af5b73e6e44b4e6f9c610450c3f618d6 P 432ac21283b94a269b47bdb8f90040b0: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260924 11:34:45.268977 15109 ts_tablet_manager.cc:1428] T af5b73e6e44b4e6f9c610450c3f618d6 P 4dddd225a8d649fabae7c12fa9fe99cc: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20260924 11:34:45.451928 15117 raft_consensus.cc:482] T af5b73e6e44b4e6f9c610450c3f618d6 P 432ac21283b94a269b47bdb8f90040b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:34:45.452179 15117 raft_consensus.cc:504] T af5b73e6e44b4e6f9c610450c3f618d6 P 432ac21283b94a269b47bdb8f90040b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } } peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } }
I20260924 11:34:45.454078 15117 leader_election.cc:288] T af5b73e6e44b4e6f9c610450c3f618d6 P 432ac21283b94a269b47bdb8f90040b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4dddd225a8d649fabae7c12fa9fe99cc (127.6.212.1:42819), 2a86a28876eb407596a0452c86e6dd45 (127.6.212.3:41957)
I20260924 11:34:45.473423 15152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:45.454633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:58003 (local address 127.6.212.3:41957)
0924 11:34:45.465882 (+ 11249us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:45.465892 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:45.467009 (+ 1117us) server_negotiation.cc:407] Connection header received
0924 11:34:45.467261 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:45.467269 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:45.467350 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:45.467488 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:34:45.468214 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:45.469317 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:45.471577 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:45.472308 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:45.472472 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:45.472575 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:45.472970 (+ 395us) server_negotiation.cc:299] Negotiation successful
0924 11:34:45.473118 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11054,"thread_start_us":11273,"threads_started":1}
I20260924 11:34:45.473475 15150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:45.454351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.3:41957 (local address 127.6.212.2:58003)
0924 11:34:45.466890 (+ 12539us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:45.466909 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:45.467108 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:45.467612 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:45.467615 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:45.467633 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:34:45.468028 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:45.468044 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:45.469469 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:45.469474 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:45.471428 (+ 1954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:45.471449 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:45.472527 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:45.472533 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:45.472742 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:45.472828 (+ 86us) client_negotiation.cc:769] Sending connection context
0924 11:34:45.472934 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 11:34:45.473088 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12423,"mutex_wait_us":255,"thread_start_us":11606,"threads_started":1}
I20260924 11:34:45.473542 15108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:45.454983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:38049 (local address 127.6.212.1:42819)
0924 11:34:45.455169 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:45.455176 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:45.466742 (+ 11566us) server_negotiation.cc:407] Connection header received
0924 11:34:45.467102 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:45.467107 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:45.467193 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:45.467288 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:34:45.467917 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:45.468739 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:45.471193 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:45.472510 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:45.472640 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:45.472823 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:45.473125 (+ 302us) server_negotiation.cc:299] Negotiation successful
0924 11:34:45.473276 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 11:34:45.474128 15151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:45.454338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:42819 (local address 127.6.212.2:38049)
0924 11:34:45.466586 (+ 12248us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:45.466613 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:45.466884 (+ 271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:45.467321 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:45.467326 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:45.467349 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:34:45.467710 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:45.467724 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:45.468866 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:45.468874 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:45.471003 (+ 2129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:45.471019 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:45.472739 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:45.472744 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:45.472960 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:45.472988 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 11:34:45.473676 (+ 688us) client_negotiation.cc:240] Negotiation successful
0924 11:34:45.473815 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12051,"mutex_wait_us":21,"thread_start_us":12237,"threads_started":1}
I20260924 11:34:45.474088 15115 raft_consensus.cc:482] T af5b73e6e44b4e6f9c610450c3f618d6 P 4dddd225a8d649fabae7c12fa9fe99cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:34:45.474272 15115 raft_consensus.cc:504] T af5b73e6e44b4e6f9c610450c3f618d6 P 4dddd225a8d649fabae7c12fa9fe99cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } } peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } }
I20260924 11:34:45.474857 14757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af5b73e6e44b4e6f9c610450c3f618d6" candidate_uuid: "432ac21283b94a269b47bdb8f90040b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" is_pre_election: true
I20260924 11:34:45.475013 15115 leader_election.cc:288] T af5b73e6e44b4e6f9c610450c3f618d6 P 4dddd225a8d649fabae7c12fa9fe99cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 432ac21283b94a269b47bdb8f90040b0 (127.6.212.2:37311), 2a86a28876eb407596a0452c86e6dd45 (127.6.212.3:41957)
I20260924 11:34:45.475003 15039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af5b73e6e44b4e6f9c610450c3f618d6" candidate_uuid: "432ac21283b94a269b47bdb8f90040b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a86a28876eb407596a0452c86e6dd45" is_pre_election: true
I20260924 11:34:45.475093 14757 raft_consensus.cc:2454] T af5b73e6e44b4e6f9c610450c3f618d6 P 4dddd225a8d649fabae7c12fa9fe99cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 432ac21283b94a269b47bdb8f90040b0 in term 0.
I20260924 11:34:45.475235 15039 raft_consensus.cc:2454] T af5b73e6e44b4e6f9c610450c3f618d6 P 2a86a28876eb407596a0452c86e6dd45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 432ac21283b94a269b47bdb8f90040b0 in term 0.
I20260924 11:34:45.475769 14827 leader_election.cc:302] T af5b73e6e44b4e6f9c610450c3f618d6 P 432ac21283b94a269b47bdb8f90040b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 432ac21283b94a269b47bdb8f90040b0, 4dddd225a8d649fabae7c12fa9fe99cc; no voters:
I20260924 11:34:45.475880 14896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af5b73e6e44b4e6f9c610450c3f618d6" candidate_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "432ac21283b94a269b47bdb8f90040b0" is_pre_election: true
I20260924 11:34:45.475937 15039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af5b73e6e44b4e6f9c610450c3f618d6" candidate_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a86a28876eb407596a0452c86e6dd45" is_pre_election: true
I20260924 11:34:45.476110 15039 raft_consensus.cc:2454] T af5b73e6e44b4e6f9c610450c3f618d6 P 2a86a28876eb407596a0452c86e6dd45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dddd225a8d649fabae7c12fa9fe99cc in term 0.
I20260924 11:34:45.476146 14896 raft_consensus.cc:2454] T af5b73e6e44b4e6f9c610450c3f618d6 P 432ac21283b94a269b47bdb8f90040b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dddd225a8d649fabae7c12fa9fe99cc in term 0.
I20260924 11:34:45.476305 15117 raft_consensus.cc:2781] T af5b73e6e44b4e6f9c610450c3f618d6 P 432ac21283b94a269b47bdb8f90040b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:34:45.476411 15117 raft_consensus.cc:482] T af5b73e6e44b4e6f9c610450c3f618d6 P 432ac21283b94a269b47bdb8f90040b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:34:45.476496 15117 raft_consensus.cc:3037] T af5b73e6e44b4e6f9c610450c3f618d6 P 432ac21283b94a269b47bdb8f90040b0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:34:45.476532 14689 leader_election.cc:302] T af5b73e6e44b4e6f9c610450c3f618d6 P 4dddd225a8d649fabae7c12fa9fe99cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a86a28876eb407596a0452c86e6dd45, 4dddd225a8d649fabae7c12fa9fe99cc; no voters:
I20260924 11:34:45.476769 15115 raft_consensus.cc:2781] T af5b73e6e44b4e6f9c610450c3f618d6 P 4dddd225a8d649fabae7c12fa9fe99cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:34:45.476881 15115 raft_consensus.cc:482] T af5b73e6e44b4e6f9c610450c3f618d6 P 4dddd225a8d649fabae7c12fa9fe99cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:34:45.476953 15115 raft_consensus.cc:3037] T af5b73e6e44b4e6f9c610450c3f618d6 P 4dddd225a8d649fabae7c12fa9fe99cc [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:34:45.479312 15115 raft_consensus.cc:504] T af5b73e6e44b4e6f9c610450c3f618d6 P 4dddd225a8d649fabae7c12fa9fe99cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } } peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } }
I20260924 11:34:45.479521 15117 raft_consensus.cc:504] T af5b73e6e44b4e6f9c610450c3f618d6 P 432ac21283b94a269b47bdb8f90040b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } } peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } }
I20260924 11:34:45.480017 15115 leader_election.cc:288] T af5b73e6e44b4e6f9c610450c3f618d6 P 4dddd225a8d649fabae7c12fa9fe99cc [CANDIDATE]: Term 1 election: Requested vote from peers 432ac21283b94a269b47bdb8f90040b0 (127.6.212.2:37311), 2a86a28876eb407596a0452c86e6dd45 (127.6.212.3:41957)
I20260924 11:34:45.480193 15117 leader_election.cc:288] T af5b73e6e44b4e6f9c610450c3f618d6 P 432ac21283b94a269b47bdb8f90040b0 [CANDIDATE]: Term 1 election: Requested vote from peers 4dddd225a8d649fabae7c12fa9fe99cc (127.6.212.1:42819), 2a86a28876eb407596a0452c86e6dd45 (127.6.212.3:41957)
I20260924 11:34:45.480784 14757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af5b73e6e44b4e6f9c610450c3f618d6" candidate_uuid: "432ac21283b94a269b47bdb8f90040b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dddd225a8d649fabae7c12fa9fe99cc"
I20260924 11:34:45.480862 15039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af5b73e6e44b4e6f9c610450c3f618d6" candidate_uuid: "432ac21283b94a269b47bdb8f90040b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a86a28876eb407596a0452c86e6dd45"
I20260924 11:34:45.480841 15038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af5b73e6e44b4e6f9c610450c3f618d6" candidate_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a86a28876eb407596a0452c86e6dd45"
I20260924 11:34:45.481051 14757 raft_consensus.cc:2379] T af5b73e6e44b4e6f9c610450c3f618d6 P 4dddd225a8d649fabae7c12fa9fe99cc [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 432ac21283b94a269b47bdb8f90040b0 in current term 1: Already voted for candidate 4dddd225a8d649fabae7c12fa9fe99cc in this term.
I20260924 11:34:45.481063 15039 raft_consensus.cc:3037] T af5b73e6e44b4e6f9c610450c3f618d6 P 2a86a28876eb407596a0452c86e6dd45 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:34:45.480356 14896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af5b73e6e44b4e6f9c610450c3f618d6" candidate_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "432ac21283b94a269b47bdb8f90040b0"
I20260924 11:34:45.481714 14896 raft_consensus.cc:2379] T af5b73e6e44b4e6f9c610450c3f618d6 P 432ac21283b94a269b47bdb8f90040b0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4dddd225a8d649fabae7c12fa9fe99cc in current term 1: Already voted for candidate 432ac21283b94a269b47bdb8f90040b0 in this term.
I20260924 11:34:45.482088 14690 leader_election.cc:302] T af5b73e6e44b4e6f9c610450c3f618d6 P 4dddd225a8d649fabae7c12fa9fe99cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4dddd225a8d649fabae7c12fa9fe99cc; no voters: 2a86a28876eb407596a0452c86e6dd45, 432ac21283b94a269b47bdb8f90040b0
I20260924 11:34:45.482487 15115 raft_consensus.cc:2726] T af5b73e6e44b4e6f9c610450c3f618d6 P 4dddd225a8d649fabae7c12fa9fe99cc [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260924 11:34:45.483428 15039 raft_consensus.cc:2454] T af5b73e6e44b4e6f9c610450c3f618d6 P 2a86a28876eb407596a0452c86e6dd45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 432ac21283b94a269b47bdb8f90040b0 in term 1.
I20260924 11:34:45.483857 14829 leader_election.cc:302] T af5b73e6e44b4e6f9c610450c3f618d6 P 432ac21283b94a269b47bdb8f90040b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a86a28876eb407596a0452c86e6dd45, 432ac21283b94a269b47bdb8f90040b0; no voters: 4dddd225a8d649fabae7c12fa9fe99cc
I20260924 11:34:45.484202 15117 raft_consensus.cc:2781] T af5b73e6e44b4e6f9c610450c3f618d6 P 432ac21283b94a269b47bdb8f90040b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:34:45.484899 15117 raft_consensus.cc:686] T af5b73e6e44b4e6f9c610450c3f618d6 P 432ac21283b94a269b47bdb8f90040b0 [term 1 LEADER]: Becoming Leader. State: Replica: 432ac21283b94a269b47bdb8f90040b0, State: Running, Role: LEADER
I20260924 11:34:45.485299 15117 consensus_queue.cc:237] T af5b73e6e44b4e6f9c610450c3f618d6 P 432ac21283b94a269b47bdb8f90040b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } } peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } }
I20260924 11:34:45.487608 15153 tablet_replica.cc:439] T af5b73e6e44b4e6f9c610450c3f618d6 P 432ac21283b94a269b47bdb8f90040b0: TxnStatusTablet state changed. Reason: New leader 432ac21283b94a269b47bdb8f90040b0. Latest consensus state: current_term: 1 leader_uuid: "432ac21283b94a269b47bdb8f90040b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } } peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } } }
I20260924 11:34:45.487979 15153 tablet_replica.cc:442] T af5b73e6e44b4e6f9c610450c3f618d6 P 432ac21283b94a269b47bdb8f90040b0: This TxnStatusTablet replica's current role is: LEADER
I20260924 11:34:45.488639 15155 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 11:34:45.488854 15155 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 11:34:45.488929 14598 catalog_manager.cc:5495] T af5b73e6e44b4e6f9c610450c3f618d6 P 432ac21283b94a269b47bdb8f90040b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 432ac21283b94a269b47bdb8f90040b0 (127.6.212.2). New cstate: current_term: 1 leader_uuid: "432ac21283b94a269b47bdb8f90040b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } health_report { overall_health: UNKNOWN } } }
I20260924 11:34:45.555230 15146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:45.548971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.2:37311 (local address 127.0.0.1:56316)
0924 11:34:45.549230 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:45.549248 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:45.549374 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:45.549897 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:45.549900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:45.549909 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 11:34:45.550068 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:45.550074 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:45.551511 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:45.551515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:45.552485 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:45.552495 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:45.553444 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:45.553448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:45.553652 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:45.553715 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 11:34:45.554720 (+ 1005us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 11:34:45.554737 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 11:34:45.554802 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 11:34:45.554957 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 11:34:45.555248 15103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:45.549113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56316 (local address 127.6.212.2:37311)
0924 11:34:45.549309 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:45.549316 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:45.549338 (+ 22us) server_negotiation.cc:407] Connection header received
0924 11:34:45.549572 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:45.549583 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:45.549689 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:45.549839 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:34:45.550196 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:45.551396 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:45.552615 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:45.553294 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:45.553428 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:45.553906 (+ 478us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:34:45.554607 (+ 701us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:34:45.554740 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:45.554849 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 11:34:45.554943 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 11:34:45.559877 14757 raft_consensus.cc:1261] T af5b73e6e44b4e6f9c610450c3f618d6 P 4dddd225a8d649fabae7c12fa9fe99cc [term 1 FOLLOWER]: Refusing update from remote peer 432ac21283b94a269b47bdb8f90040b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:34:45.559899 15039 raft_consensus.cc:1261] T af5b73e6e44b4e6f9c610450c3f618d6 P 2a86a28876eb407596a0452c86e6dd45 [term 1 FOLLOWER]: Refusing update from remote peer 432ac21283b94a269b47bdb8f90040b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:34:45.560683 15153 consensus_queue.cc:1035] T af5b73e6e44b4e6f9c610450c3f618d6 P 432ac21283b94a269b47bdb8f90040b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:34:45.561033 15117 consensus_queue.cc:1035] T af5b73e6e44b4e6f9c610450c3f618d6 P 432ac21283b94a269b47bdb8f90040b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:34:45.568281 15115 tablet_replica.cc:439] T af5b73e6e44b4e6f9c610450c3f618d6 P 4dddd225a8d649fabae7c12fa9fe99cc: TxnStatusTablet state changed. Reason: New leader 432ac21283b94a269b47bdb8f90040b0. Latest consensus state: current_term: 1 leader_uuid: "432ac21283b94a269b47bdb8f90040b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } } peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } } }
I20260924 11:34:45.568591 15115 tablet_replica.cc:442] T af5b73e6e44b4e6f9c610450c3f618d6 P 4dddd225a8d649fabae7c12fa9fe99cc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:34:45.572726 15116 tablet_replica.cc:439] T af5b73e6e44b4e6f9c610450c3f618d6 P 2a86a28876eb407596a0452c86e6dd45: TxnStatusTablet state changed. Reason: New leader 432ac21283b94a269b47bdb8f90040b0. Latest consensus state: current_term: 1 leader_uuid: "432ac21283b94a269b47bdb8f90040b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } } peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } } }
I20260924 11:34:45.573009 15116 tablet_replica.cc:442] T af5b73e6e44b4e6f9c610450c3f618d6 P 2a86a28876eb407596a0452c86e6dd45: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:34:45.579455 15115 tablet_replica.cc:439] T af5b73e6e44b4e6f9c610450c3f618d6 P 4dddd225a8d649fabae7c12fa9fe99cc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "432ac21283b94a269b47bdb8f90040b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } } peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } } }
I20260924 11:34:45.579727 15115 tablet_replica.cc:442] T af5b73e6e44b4e6f9c610450c3f618d6 P 4dddd225a8d649fabae7c12fa9fe99cc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:34:45.582226 15116 tablet_replica.cc:439] T af5b73e6e44b4e6f9c610450c3f618d6 P 2a86a28876eb407596a0452c86e6dd45: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "432ac21283b94a269b47bdb8f90040b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } } peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } } }
I20260924 11:34:45.582547 15116 tablet_replica.cc:442] T af5b73e6e44b4e6f9c610450c3f618d6 P 2a86a28876eb407596a0452c86e6dd45: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 11:34:45.586162 15166 tablet_replica.cc:439] T af5b73e6e44b4e6f9c610450c3f618d6 P 432ac21283b94a269b47bdb8f90040b0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "432ac21283b94a269b47bdb8f90040b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } } peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } } }
I20260924 11:34:45.586582 15166 tablet_replica.cc:442] T af5b73e6e44b4e6f9c610450c3f618d6 P 432ac21283b94a269b47bdb8f90040b0: This TxnStatusTablet replica's current role is: LEADER
I20260924 11:34:45.586755 15153 tablet_replica.cc:439] T af5b73e6e44b4e6f9c610450c3f618d6 P 432ac21283b94a269b47bdb8f90040b0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "432ac21283b94a269b47bdb8f90040b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } } peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } } }
I20260924 11:34:45.587013 15153 tablet_replica.cc:442] T af5b73e6e44b4e6f9c610450c3f618d6 P 432ac21283b94a269b47bdb8f90040b0: This TxnStatusTablet replica's current role is: LEADER
I20260924 11:34:45.598116 15152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:45.591603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50386 (local address 127.6.212.3:41957)
0924 11:34:45.591842 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:45.591850 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:45.592190 (+ 340us) server_negotiation.cc:407] Connection header received
0924 11:34:45.592413 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:45.592418 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:45.592502 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:45.592624 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:34:45.593091 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:45.593754 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:45.595682 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:45.596274 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:45.596377 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:45.596922 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:34:45.597521 (+ 599us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:34:45.597635 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:45.597763 (+ 128us) server_negotiation.cc:299] Negotiation successful
0924 11:34:45.597825 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 11:34:45.614965 15171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:45.608738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.2:37311 (local address 127.6.212.3:56447)
0924 11:34:45.609575 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:45.609594 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:45.609714 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:45.610052 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:45.610055 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:45.610064 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 11:34:45.610309 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:45.610316 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:45.611788 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:45.611792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:45.612929 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:45.612941 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:45.613928 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:45.613932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:45.614050 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:45.614103 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 11:34:45.614522 (+ 419us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 11:34:45.614530 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 11:34:45.614588 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 11:34:45.614693 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":528,"threads_started":1}
I20260924 11:34:45.615017 15103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:45.608802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:56447 (local address 127.6.212.2:37311)
0924 11:34:45.609045 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:45.609066 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:45.609681 (+ 615us) server_negotiation.cc:407] Connection header received
0924 11:34:45.609835 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:45.609840 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:45.609933 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:45.610033 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:34:45.610501 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:45.611663 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:45.613065 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:45.613761 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:45.613887 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:45.614276 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:34:45.614419 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:34:45.614518 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:45.614635 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 11:34:45.614717 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 11:34:45.624439 14896 raft_consensus.cc:1261] T e6e9ef978ee048979379e976c422f03b P 432ac21283b94a269b47bdb8f90040b0 [term 1 FOLLOWER]: Refusing update from remote peer 2a86a28876eb407596a0452c86e6dd45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:34:45.624451 14757 raft_consensus.cc:1261] T e6e9ef978ee048979379e976c422f03b P 4dddd225a8d649fabae7c12fa9fe99cc [term 1 FOLLOWER]: Refusing update from remote peer 2a86a28876eb407596a0452c86e6dd45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:34:45.625082 15116 consensus_queue.cc:1035] T e6e9ef978ee048979379e976c422f03b P 2a86a28876eb407596a0452c86e6dd45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:34:45.625346 15139 consensus_queue.cc:1035] T e6e9ef978ee048979379e976c422f03b P 2a86a28876eb407596a0452c86e6dd45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:34:45.630189 15159 mvcc.cc:205] Tried to move back new op lower bound from 7332862712311676928 to 7332862710409379840. Current Snapshot: MvccSnapshot[applied={T|T < 7332862712311676928}]
I20260924 11:34:45.664165 15152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:45.657143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:50827 (local address 127.6.212.3:41957)
0924 11:34:45.657360 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:45.657369 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:45.657939 (+ 570us) server_negotiation.cc:407] Connection header received
0924 11:34:45.658198 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:45.658204 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:45.658289 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:45.658378 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:34:45.659400 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:45.660117 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:45.662348 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:45.662850 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:45.662943 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:45.663494 (+ 551us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:34:45.663598 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:34:45.663664 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:45.663819 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 11:34:45.663943 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 11:34:45.664356 15177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:45.657066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.3:41957 (local address 127.6.212.2:50827)
0924 11:34:45.657841 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:45.657856 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:45.658043 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:45.658528 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:45.658535 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:45.658556 (+ 21us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 11:34:45.659184 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:45.659199 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:45.660238 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:45.660245 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:45.662182 (+ 1937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:45.662197 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:45.663037 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:45.663048 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:45.663230 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:45.663301 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 11:34:45.663691 (+ 390us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 11:34:45.663702 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 11:34:45.663786 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 11:34:45.663934 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":478,"threads_started":1}
I20260924 11:34:45.701337 14600 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:34:45.701606 14600 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:34:45.705057 14600 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c9bf26035dc4b5e9edde52527a113e7: Sending DeleteTablet for 3 replicas of tablet e6e9ef978ee048979379e976c422f03b
I20260924 11:34:45.706007 15015 tablet_service.cc:1514] Processing DeleteTablet for tablet e6e9ef978ee048979379e976c422f03b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:34:45 UTC) from {username='oryx', principal='oryx/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:50376
I20260924 11:34:45.706204 14875 tablet_service.cc:1514] Processing DeleteTablet for tablet e6e9ef978ee048979379e976c422f03b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:34:45 UTC) from {username='oryx', principal='oryx/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:56300
I20260924 11:34:45.717149 14737 tablet_service.cc:1514] Processing DeleteTablet for tablet e6e9ef978ee048979379e976c422f03b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:34:45 UTC) from {username='oryx', principal='oryx/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:52550
I20260924 11:34:45.728675 15180 tablet_replica.cc:330] T e6e9ef978ee048979379e976c422f03b P 2a86a28876eb407596a0452c86e6dd45: stopping tablet replica
I20260924 11:34:45.729074 15180 raft_consensus.cc:2229] T e6e9ef978ee048979379e976c422f03b P 2a86a28876eb407596a0452c86e6dd45 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:34:45.729164 15182 tablet_replica.cc:330] T e6e9ef978ee048979379e976c422f03b P 4dddd225a8d649fabae7c12fa9fe99cc: stopping tablet replica
I20260924 11:34:45.729352 15182 raft_consensus.cc:2229] T e6e9ef978ee048979379e976c422f03b P 4dddd225a8d649fabae7c12fa9fe99cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:34:45.729581 15182 raft_consensus.cc:2258] T e6e9ef978ee048979379e976c422f03b P 4dddd225a8d649fabae7c12fa9fe99cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:34:45.729928 15180 raft_consensus.cc:2258] T e6e9ef978ee048979379e976c422f03b P 2a86a28876eb407596a0452c86e6dd45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:34:45.731107 15180 ts_tablet_manager.cc:1905] T e6e9ef978ee048979379e976c422f03b P 2a86a28876eb407596a0452c86e6dd45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:34:45.737999 15182 ts_tablet_manager.cc:1905] T e6e9ef978ee048979379e976c422f03b P 4dddd225a8d649fabae7c12fa9fe99cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:34:45.738643 15181 tablet_replica.cc:330] T e6e9ef978ee048979379e976c422f03b P 432ac21283b94a269b47bdb8f90040b0: stopping tablet replica
I20260924 11:34:45.738951 15181 raft_consensus.cc:2229] T e6e9ef978ee048979379e976c422f03b P 432ac21283b94a269b47bdb8f90040b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:34:45.743479 15181 raft_consensus.cc:2258] T e6e9ef978ee048979379e976c422f03b P 432ac21283b94a269b47bdb8f90040b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:34:45.744772 15181 ts_tablet_manager.cc:1905] T e6e9ef978ee048979379e976c422f03b P 432ac21283b94a269b47bdb8f90040b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:34:45.745221 15182 ts_tablet_manager.cc:1918] T e6e9ef978ee048979379e976c422f03b P 4dddd225a8d649fabae7c12fa9fe99cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 11:34:45.755473 15182 log.cc:1196] T e6e9ef978ee048979379e976c422f03b P 4dddd225a8d649fabae7c12fa9fe99cc: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-0/wal/wals/e6e9ef978ee048979379e976c422f03b
I20260924 11:34:45.756095 15182 ts_tablet_manager.cc:1939] T e6e9ef978ee048979379e976c422f03b P 4dddd225a8d649fabae7c12fa9fe99cc: Deleting consensus metadata
I20260924 11:34:45.757273 14584 catalog_manager.cc:4841] TS 4dddd225a8d649fabae7c12fa9fe99cc (127.6.212.1:42819): tablet e6e9ef978ee048979379e976c422f03b (table test-table [id=1d3fc936fac14f2aa1e3cec05baafd97]) successfully deleted
I20260924 11:34:45.762042 15180 ts_tablet_manager.cc:1918] T e6e9ef978ee048979379e976c422f03b P 2a86a28876eb407596a0452c86e6dd45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 11:34:45.767484 15180 log.cc:1196] T e6e9ef978ee048979379e976c422f03b P 2a86a28876eb407596a0452c86e6dd45: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-2/wal/wals/e6e9ef978ee048979379e976c422f03b
I20260924 11:34:45.768157 15180 ts_tablet_manager.cc:1939] T e6e9ef978ee048979379e976c422f03b P 2a86a28876eb407596a0452c86e6dd45: Deleting consensus metadata
I20260924 11:34:45.769451 14586 catalog_manager.cc:4841] TS 2a86a28876eb407596a0452c86e6dd45 (127.6.212.3:41957): tablet e6e9ef978ee048979379e976c422f03b (table test-table [id=1d3fc936fac14f2aa1e3cec05baafd97]) successfully deleted
I20260924 11:34:45.770780 15181 ts_tablet_manager.cc:1918] T e6e9ef978ee048979379e976c422f03b P 432ac21283b94a269b47bdb8f90040b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 11:34:45.771005 15181 log.cc:1196] T e6e9ef978ee048979379e976c422f03b P 432ac21283b94a269b47bdb8f90040b0: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-1/wal/wals/e6e9ef978ee048979379e976c422f03b
I20260924 11:34:45.771576 15181 ts_tablet_manager.cc:1939] T e6e9ef978ee048979379e976c422f03b P 432ac21283b94a269b47bdb8f90040b0: Deleting consensus metadata
I20260924 11:34:45.772837 14587 catalog_manager.cc:4841] TS 432ac21283b94a269b47bdb8f90040b0 (127.6.212.2:37311): tablet e6e9ef978ee048979379e976c422f03b (table test-table [id=1d3fc936fac14f2aa1e3cec05baafd97]) successfully deleted
Sep 24 11:34:45 dist-test-slave-cq7c krb5kdc[14555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 11:34:45 dist-test-slave-cq7c krb5kdc[14555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:34:45 dist-test-slave-cq7c krb5kdc[14555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 11:34:45.848953 15190 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:34:45.821911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43408 (local address 127.6.212.62:39467)
0924 11:34:45.835838 (+ 13927us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:45.835845 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:45.836255 (+ 410us) server_negotiation.cc:407] Connection header received
0924 11:34:45.836461 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:45.836466 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:45.836559 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:45.836695 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:45.838536 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:45.839660 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:45.841560 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:45.842264 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:45.842406 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:45.848608 (+ 6202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43408: BlockingRecv error: recv got EOF from 127.0.0.1:43408 (error 108)
Metrics: {"server-negotiator.queue_time_us":13760,"thread_start_us":13617,"threads_started":1}
Sep 24 11:34:45 dist-test-slave-cq7c krb5kdc[14555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM
I20260924 11:34:45.871444 15190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:45.852935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43416 (local address 127.6.212.62:39467)
0924 11:34:45.853107 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:45.853115 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:45.853594 (+ 479us) server_negotiation.cc:407] Connection header received
0924 11:34:45.853780 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:45.853786 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:45.853884 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:45.854014 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:45.855708 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:45.856875 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:45.858764 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:45.859451 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:45.859600 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:45.864854 (+ 5254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:45.864863 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:45.864868 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:45.864915 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:45.868400 (+ 3485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:45.869244 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:45.869249 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:45.869253 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:45.869416 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:45.869859 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:45.869864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:45.869868 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:45.870200 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:45.870441 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:45.870806 (+ 365us) server_negotiation.cc:299] Negotiation successful
0924 11:34:45.871043 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 11:34:45.875620 14600 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43416:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:34:45.876374 14600 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:34:45.884918 14737 tablet_service.cc:1467] Processing CreateTablet for tablet 2d31fa27069e401483d3f0d3a72e17c5 (DEFAULT_TABLE table=test-table [id=e44bce1496da458c806699a73b72580b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:34:45.885421 14737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d31fa27069e401483d3f0d3a72e17c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:34:45.885450 14875 tablet_service.cc:1467] Processing CreateTablet for tablet 2d31fa27069e401483d3f0d3a72e17c5 (DEFAULT_TABLE table=test-table [id=e44bce1496da458c806699a73b72580b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:34:45.885934 14875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d31fa27069e401483d3f0d3a72e17c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:34:45.891882 15015 tablet_service.cc:1467] Processing CreateTablet for tablet 2d31fa27069e401483d3f0d3a72e17c5 (DEFAULT_TABLE table=test-table [id=e44bce1496da458c806699a73b72580b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:34:45.892339 15015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d31fa27069e401483d3f0d3a72e17c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:34:45.897957 15196 tablet_bootstrap.cc:492] T 2d31fa27069e401483d3f0d3a72e17c5 P 4dddd225a8d649fabae7c12fa9fe99cc: Bootstrap starting.
I20260924 11:34:45.900305 15196 tablet_bootstrap.cc:654] T 2d31fa27069e401483d3f0d3a72e17c5 P 4dddd225a8d649fabae7c12fa9fe99cc: Neither blocks nor log segments found. Creating new log.
I20260924 11:34:45.912868 15196 tablet_bootstrap.cc:492] T 2d31fa27069e401483d3f0d3a72e17c5 P 4dddd225a8d649fabae7c12fa9fe99cc: No bootstrap required, opened a new log
I20260924 11:34:45.913137 15196 ts_tablet_manager.cc:1397] T 2d31fa27069e401483d3f0d3a72e17c5 P 4dddd225a8d649fabae7c12fa9fe99cc: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.010s
I20260924 11:34:45.914021 15196 raft_consensus.cc:348] T 2d31fa27069e401483d3f0d3a72e17c5 P 4dddd225a8d649fabae7c12fa9fe99cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } } peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } }
I20260924 11:34:45.914433 15196 raft_consensus.cc:374] T 2d31fa27069e401483d3f0d3a72e17c5 P 4dddd225a8d649fabae7c12fa9fe99cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:34:45.914564 15196 raft_consensus.cc:729] T 2d31fa27069e401483d3f0d3a72e17c5 P 4dddd225a8d649fabae7c12fa9fe99cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dddd225a8d649fabae7c12fa9fe99cc, State: Initialized, Role: FOLLOWER
I20260924 11:34:45.915087 15196 consensus_queue.cc:260] T 2d31fa27069e401483d3f0d3a72e17c5 P 4dddd225a8d649fabae7c12fa9fe99cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } } peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } }
I20260924 11:34:45.918720 15196 ts_tablet_manager.cc:1428] T 2d31fa27069e401483d3f0d3a72e17c5 P 4dddd225a8d649fabae7c12fa9fe99cc: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20260924 11:34:45.921465 15199 tablet_bootstrap.cc:492] T 2d31fa27069e401483d3f0d3a72e17c5 P 432ac21283b94a269b47bdb8f90040b0: Bootstrap starting.
I20260924 11:34:45.923743 15199 tablet_bootstrap.cc:654] T 2d31fa27069e401483d3f0d3a72e17c5 P 432ac21283b94a269b47bdb8f90040b0: Neither blocks nor log segments found. Creating new log.
I20260924 11:34:45.925887 15198 tablet_bootstrap.cc:492] T 2d31fa27069e401483d3f0d3a72e17c5 P 2a86a28876eb407596a0452c86e6dd45: Bootstrap starting.
I20260924 11:34:45.928026 15198 tablet_bootstrap.cc:654] T 2d31fa27069e401483d3f0d3a72e17c5 P 2a86a28876eb407596a0452c86e6dd45: Neither blocks nor log segments found. Creating new log.
I20260924 11:34:45.930821 15199 tablet_bootstrap.cc:492] T 2d31fa27069e401483d3f0d3a72e17c5 P 432ac21283b94a269b47bdb8f90040b0: No bootstrap required, opened a new log
I20260924 11:34:45.931042 15199 ts_tablet_manager.cc:1397] T 2d31fa27069e401483d3f0d3a72e17c5 P 432ac21283b94a269b47bdb8f90040b0: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260924 11:34:45.931085 15198 tablet_bootstrap.cc:492] T 2d31fa27069e401483d3f0d3a72e17c5 P 2a86a28876eb407596a0452c86e6dd45: No bootstrap required, opened a new log
I20260924 11:34:45.931310 15198 ts_tablet_manager.cc:1397] T 2d31fa27069e401483d3f0d3a72e17c5 P 2a86a28876eb407596a0452c86e6dd45: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 11:34:45.931978 15199 raft_consensus.cc:348] T 2d31fa27069e401483d3f0d3a72e17c5 P 432ac21283b94a269b47bdb8f90040b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } } peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } }
I20260924 11:34:45.932307 15199 raft_consensus.cc:374] T 2d31fa27069e401483d3f0d3a72e17c5 P 432ac21283b94a269b47bdb8f90040b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:34:45.932184 15198 raft_consensus.cc:348] T 2d31fa27069e401483d3f0d3a72e17c5 P 2a86a28876eb407596a0452c86e6dd45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } } peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } }
I20260924 11:34:45.932459 15199 raft_consensus.cc:729] T 2d31fa27069e401483d3f0d3a72e17c5 P 432ac21283b94a269b47bdb8f90040b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 432ac21283b94a269b47bdb8f90040b0, State: Initialized, Role: FOLLOWER
I20260924 11:34:45.932530 15198 raft_consensus.cc:374] T 2d31fa27069e401483d3f0d3a72e17c5 P 2a86a28876eb407596a0452c86e6dd45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:34:45.932679 15198 raft_consensus.cc:729] T 2d31fa27069e401483d3f0d3a72e17c5 P 2a86a28876eb407596a0452c86e6dd45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a86a28876eb407596a0452c86e6dd45, State: Initialized, Role: FOLLOWER
I20260924 11:34:45.932823 15199 consensus_queue.cc:260] T 2d31fa27069e401483d3f0d3a72e17c5 P 432ac21283b94a269b47bdb8f90040b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } } peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } }
I20260924 11:34:45.934046 15198 consensus_queue.cc:260] T 2d31fa27069e401483d3f0d3a72e17c5 P 2a86a28876eb407596a0452c86e6dd45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } } peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } }
I20260924 11:34:45.935851 15199 ts_tablet_manager.cc:1428] T 2d31fa27069e401483d3f0d3a72e17c5 P 432ac21283b94a269b47bdb8f90040b0: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260924 11:34:45.936910 15198 ts_tablet_manager.cc:1428] T 2d31fa27069e401483d3f0d3a72e17c5 P 2a86a28876eb407596a0452c86e6dd45: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260924 11:34:46.071563 15115 raft_consensus.cc:482] T 2d31fa27069e401483d3f0d3a72e17c5 P 4dddd225a8d649fabae7c12fa9fe99cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:34:46.071805 15115 raft_consensus.cc:504] T 2d31fa27069e401483d3f0d3a72e17c5 P 4dddd225a8d649fabae7c12fa9fe99cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } } peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } }
I20260924 11:34:46.072472 15115 leader_election.cc:288] T 2d31fa27069e401483d3f0d3a72e17c5 P 4dddd225a8d649fabae7c12fa9fe99cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 432ac21283b94a269b47bdb8f90040b0 (127.6.212.2:37311), 2a86a28876eb407596a0452c86e6dd45 (127.6.212.3:41957)
I20260924 11:34:46.073088 14896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d31fa27069e401483d3f0d3a72e17c5" candidate_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "432ac21283b94a269b47bdb8f90040b0" is_pre_election: true
I20260924 11:34:46.073137 15039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d31fa27069e401483d3f0d3a72e17c5" candidate_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a86a28876eb407596a0452c86e6dd45" is_pre_election: true
I20260924 11:34:46.073347 14896 raft_consensus.cc:2454] T 2d31fa27069e401483d3f0d3a72e17c5 P 432ac21283b94a269b47bdb8f90040b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dddd225a8d649fabae7c12fa9fe99cc in term 0.
I20260924 11:34:46.073354 15039 raft_consensus.cc:2454] T 2d31fa27069e401483d3f0d3a72e17c5 P 2a86a28876eb407596a0452c86e6dd45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dddd225a8d649fabae7c12fa9fe99cc in term 0.
I20260924 11:34:46.073845 14690 leader_election.cc:302] T 2d31fa27069e401483d3f0d3a72e17c5 P 4dddd225a8d649fabae7c12fa9fe99cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2a86a28876eb407596a0452c86e6dd45, 432ac21283b94a269b47bdb8f90040b0, 4dddd225a8d649fabae7c12fa9fe99cc; no voters:
I20260924 11:34:46.074164 15115 raft_consensus.cc:2781] T 2d31fa27069e401483d3f0d3a72e17c5 P 4dddd225a8d649fabae7c12fa9fe99cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:34:46.074265 15115 raft_consensus.cc:482] T 2d31fa27069e401483d3f0d3a72e17c5 P 4dddd225a8d649fabae7c12fa9fe99cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:34:46.074386 15115 raft_consensus.cc:3037] T 2d31fa27069e401483d3f0d3a72e17c5 P 4dddd225a8d649fabae7c12fa9fe99cc [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:34:46.076488 15115 raft_consensus.cc:504] T 2d31fa27069e401483d3f0d3a72e17c5 P 4dddd225a8d649fabae7c12fa9fe99cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } } peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } }
I20260924 11:34:46.077086 15115 leader_election.cc:288] T 2d31fa27069e401483d3f0d3a72e17c5 P 4dddd225a8d649fabae7c12fa9fe99cc [CANDIDATE]: Term 1 election: Requested vote from peers 432ac21283b94a269b47bdb8f90040b0 (127.6.212.2:37311), 2a86a28876eb407596a0452c86e6dd45 (127.6.212.3:41957)
I20260924 11:34:46.077525 14896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d31fa27069e401483d3f0d3a72e17c5" candidate_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "432ac21283b94a269b47bdb8f90040b0"
I20260924 11:34:46.077612 15039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d31fa27069e401483d3f0d3a72e17c5" candidate_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a86a28876eb407596a0452c86e6dd45"
I20260924 11:34:46.077744 14896 raft_consensus.cc:3037] T 2d31fa27069e401483d3f0d3a72e17c5 P 432ac21283b94a269b47bdb8f90040b0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:34:46.077876 15039 raft_consensus.cc:3037] T 2d31fa27069e401483d3f0d3a72e17c5 P 2a86a28876eb407596a0452c86e6dd45 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:34:46.080835 14896 raft_consensus.cc:2454] T 2d31fa27069e401483d3f0d3a72e17c5 P 432ac21283b94a269b47bdb8f90040b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dddd225a8d649fabae7c12fa9fe99cc in term 1.
I20260924 11:34:46.081120 15039 raft_consensus.cc:2454] T 2d31fa27069e401483d3f0d3a72e17c5 P 2a86a28876eb407596a0452c86e6dd45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dddd225a8d649fabae7c12fa9fe99cc in term 1.
I20260924 11:34:46.081331 14690 leader_election.cc:302] T 2d31fa27069e401483d3f0d3a72e17c5 P 4dddd225a8d649fabae7c12fa9fe99cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 432ac21283b94a269b47bdb8f90040b0, 4dddd225a8d649fabae7c12fa9fe99cc; no voters:
I20260924 11:34:46.081655 15115 raft_consensus.cc:2781] T 2d31fa27069e401483d3f0d3a72e17c5 P 4dddd225a8d649fabae7c12fa9fe99cc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:34:46.082250 15115 raft_consensus.cc:686] T 2d31fa27069e401483d3f0d3a72e17c5 P 4dddd225a8d649fabae7c12fa9fe99cc [term 1 LEADER]: Becoming Leader. State: Replica: 4dddd225a8d649fabae7c12fa9fe99cc, State: Running, Role: LEADER
I20260924 11:34:46.082566 15115 consensus_queue.cc:237] T 2d31fa27069e401483d3f0d3a72e17c5 P 4dddd225a8d649fabae7c12fa9fe99cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } } peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } }
I20260924 11:34:46.085754 14598 catalog_manager.cc:5495] T 2d31fa27069e401483d3f0d3a72e17c5 P 4dddd225a8d649fabae7c12fa9fe99cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 4dddd225a8d649fabae7c12fa9fe99cc (127.6.212.1). New cstate: current_term: 1 leader_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } health_report { overall_health: UNKNOWN } } }
I20260924 11:34:46.106469 15205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:46.100505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52570 (local address 127.6.212.1:42819)
0924 11:34:46.101113 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:46.101119 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:46.101156 (+ 37us) server_negotiation.cc:407] Connection header received
0924 11:34:46.101224 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:46.101229 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:46.101313 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:46.101435 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:34:46.101781 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:46.103011 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:46.104450 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:46.105114 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:46.105247 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:46.105680 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:34:46.105821 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:34:46.105918 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:46.106054 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 11:34:46.106134 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":316,"threads_started":1}
I20260924 11:34:46.109033 15039 raft_consensus.cc:1261] T 2d31fa27069e401483d3f0d3a72e17c5 P 2a86a28876eb407596a0452c86e6dd45 [term 1 FOLLOWER]: Refusing update from remote peer 4dddd225a8d649fabae7c12fa9fe99cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:34:46.109026 14896 raft_consensus.cc:1261] T 2d31fa27069e401483d3f0d3a72e17c5 P 432ac21283b94a269b47bdb8f90040b0 [term 1 FOLLOWER]: Refusing update from remote peer 4dddd225a8d649fabae7c12fa9fe99cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:34:46.109802 15115 consensus_queue.cc:1035] T 2d31fa27069e401483d3f0d3a72e17c5 P 4dddd225a8d649fabae7c12fa9fe99cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:34:46.110154 15202 consensus_queue.cc:1035] T 2d31fa27069e401483d3f0d3a72e17c5 P 4dddd225a8d649fabae7c12fa9fe99cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:34:46.122812 14600 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:34:46.123129 14600 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:34:46.127460 14600 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c9bf26035dc4b5e9edde52527a113e7: Sending DeleteTablet for 3 replicas of tablet 2d31fa27069e401483d3f0d3a72e17c5
I20260924 11:34:46.128329 14737 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d31fa27069e401483d3f0d3a72e17c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:34:46 UTC) from {username='oryx', principal='oryx/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:52550
I20260924 11:34:46.128454 14875 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d31fa27069e401483d3f0d3a72e17c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:34:46 UTC) from {username='oryx', principal='oryx/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:56300
I20260924 11:34:46.128611 15182 tablet_replica.cc:330] T 2d31fa27069e401483d3f0d3a72e17c5 P 4dddd225a8d649fabae7c12fa9fe99cc: stopping tablet replica
I20260924 11:34:46.128700 14876 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d31fa27069e401483d3f0d3a72e17c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:34:46 UTC) from {username='oryx', principal='oryx/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:56300
I20260924 11:34:46.128712 15181 tablet_replica.cc:330] T 2d31fa27069e401483d3f0d3a72e17c5 P 432ac21283b94a269b47bdb8f90040b0: stopping tablet replica
I20260924 11:34:46.128808 15182 raft_consensus.cc:2229] T 2d31fa27069e401483d3f0d3a72e17c5 P 4dddd225a8d649fabae7c12fa9fe99cc [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:34:46.128983 15181 raft_consensus.cc:2229] T 2d31fa27069e401483d3f0d3a72e17c5 P 432ac21283b94a269b47bdb8f90040b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:34:46.129096 15182 raft_consensus.cc:2258] T 2d31fa27069e401483d3f0d3a72e17c5 P 4dddd225a8d649fabae7c12fa9fe99cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:34:46.129165 15181 raft_consensus.cc:2258] T 2d31fa27069e401483d3f0d3a72e17c5 P 432ac21283b94a269b47bdb8f90040b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:34:46.130204 15015 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d31fa27069e401483d3f0d3a72e17c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:34:46 UTC) from {username='oryx', principal='oryx/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:50376
I20260924 11:34:46.130476 15180 tablet_replica.cc:330] T 2d31fa27069e401483d3f0d3a72e17c5 P 2a86a28876eb407596a0452c86e6dd45: stopping tablet replica
I20260924 11:34:46.130555 15181 ts_tablet_manager.cc:1905] T 2d31fa27069e401483d3f0d3a72e17c5 P 432ac21283b94a269b47bdb8f90040b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:34:46.130681 15180 raft_consensus.cc:2229] T 2d31fa27069e401483d3f0d3a72e17c5 P 2a86a28876eb407596a0452c86e6dd45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:34:46.130908 15180 raft_consensus.cc:2258] T 2d31fa27069e401483d3f0d3a72e17c5 P 2a86a28876eb407596a0452c86e6dd45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:34:46.131016 14598 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43416:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:34:46.131672 14598 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:34:46.133852 15180 ts_tablet_manager.cc:1905] T 2d31fa27069e401483d3f0d3a72e17c5 P 2a86a28876eb407596a0452c86e6dd45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:34:46.136539 15182 ts_tablet_manager.cc:1905] T 2d31fa27069e401483d3f0d3a72e17c5 P 4dddd225a8d649fabae7c12fa9fe99cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:34:46.137210 15181 ts_tablet_manager.cc:1918] T 2d31fa27069e401483d3f0d3a72e17c5 P 432ac21283b94a269b47bdb8f90040b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:34:46.137351 15181 log.cc:1196] T 2d31fa27069e401483d3f0d3a72e17c5 P 432ac21283b94a269b47bdb8f90040b0: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-1/wal/wals/2d31fa27069e401483d3f0d3a72e17c5
I20260924 11:34:46.137820 15181 ts_tablet_manager.cc:1939] T 2d31fa27069e401483d3f0d3a72e17c5 P 432ac21283b94a269b47bdb8f90040b0: Deleting consensus metadata
I20260924 11:34:46.138885 14587 catalog_manager.cc:4841] TS 432ac21283b94a269b47bdb8f90040b0 (127.6.212.2:37311): tablet 2d31fa27069e401483d3f0d3a72e17c5 (table test-table [id=e44bce1496da458c806699a73b72580b]) successfully deleted
W20260924 11:34:46.139813 14587 catalog_manager.cc:4808] TS 432ac21283b94a269b47bdb8f90040b0 (127.6.212.2:37311): delete failed for tablet 2d31fa27069e401483d3f0d3a72e17c5 because the tablet was not found. No further retry: Not found: Tablet not found: 2d31fa27069e401483d3f0d3a72e17c5
I20260924 11:34:46.140800 15180 ts_tablet_manager.cc:1918] T 2d31fa27069e401483d3f0d3a72e17c5 P 2a86a28876eb407596a0452c86e6dd45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:34:46.141000 15180 log.cc:1196] T 2d31fa27069e401483d3f0d3a72e17c5 P 2a86a28876eb407596a0452c86e6dd45: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-2/wal/wals/2d31fa27069e401483d3f0d3a72e17c5
I20260924 11:34:46.141405 15180 ts_tablet_manager.cc:1939] T 2d31fa27069e401483d3f0d3a72e17c5 P 2a86a28876eb407596a0452c86e6dd45: Deleting consensus metadata
I20260924 11:34:46.142341 14586 catalog_manager.cc:4841] TS 2a86a28876eb407596a0452c86e6dd45 (127.6.212.3:41957): tablet 2d31fa27069e401483d3f0d3a72e17c5 (table test-table [id=e44bce1496da458c806699a73b72580b]) successfully deleted
I20260924 11:34:46.143149 15182 ts_tablet_manager.cc:1918] T 2d31fa27069e401483d3f0d3a72e17c5 P 4dddd225a8d649fabae7c12fa9fe99cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:34:46.143312 15182 log.cc:1196] T 2d31fa27069e401483d3f0d3a72e17c5 P 4dddd225a8d649fabae7c12fa9fe99cc: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-0/wal/wals/2d31fa27069e401483d3f0d3a72e17c5
I20260924 11:34:46.143756 15182 ts_tablet_manager.cc:1939] T 2d31fa27069e401483d3f0d3a72e17c5 P 4dddd225a8d649fabae7c12fa9fe99cc: Deleting consensus metadata
I20260924 11:34:46.144989 14584 catalog_manager.cc:4841] TS 4dddd225a8d649fabae7c12fa9fe99cc (127.6.212.1:42819): tablet 2d31fa27069e401483d3f0d3a72e17c5 (table test-table [id=e44bce1496da458c806699a73b72580b]) successfully deleted
I20260924 11:34:46.149355 14737 tablet_service.cc:1467] Processing CreateTablet for tablet c101c1bc9aa94545b65519c2aba9e6b9 (DEFAULT_TABLE table=test-table [id=9f9439de7a19460a982e5cb6390d26ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:34:46.149375 14876 tablet_service.cc:1467] Processing CreateTablet for tablet c101c1bc9aa94545b65519c2aba9e6b9 (DEFAULT_TABLE table=test-table [id=9f9439de7a19460a982e5cb6390d26ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:34:46.149793 14876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c101c1bc9aa94545b65519c2aba9e6b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:34:46.149812 14737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c101c1bc9aa94545b65519c2aba9e6b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:34:46.156150 15015 tablet_service.cc:1467] Processing CreateTablet for tablet c101c1bc9aa94545b65519c2aba9e6b9 (DEFAULT_TABLE table=test-table [id=9f9439de7a19460a982e5cb6390d26ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:34:46.156618 15015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c101c1bc9aa94545b65519c2aba9e6b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:34:46.156804 15196 tablet_bootstrap.cc:492] T c101c1bc9aa94545b65519c2aba9e6b9 P 4dddd225a8d649fabae7c12fa9fe99cc: Bootstrap starting.
I20260924 11:34:46.158481 15196 tablet_bootstrap.cc:654] T c101c1bc9aa94545b65519c2aba9e6b9 P 4dddd225a8d649fabae7c12fa9fe99cc: Neither blocks nor log segments found. Creating new log.
I20260924 11:34:46.159643 15199 tablet_bootstrap.cc:492] T c101c1bc9aa94545b65519c2aba9e6b9 P 432ac21283b94a269b47bdb8f90040b0: Bootstrap starting.
I20260924 11:34:46.161248 15199 tablet_bootstrap.cc:654] T c101c1bc9aa94545b65519c2aba9e6b9 P 432ac21283b94a269b47bdb8f90040b0: Neither blocks nor log segments found. Creating new log.
I20260924 11:34:46.161448 15196 tablet_bootstrap.cc:492] T c101c1bc9aa94545b65519c2aba9e6b9 P 4dddd225a8d649fabae7c12fa9fe99cc: No bootstrap required, opened a new log
I20260924 11:34:46.161629 15196 ts_tablet_manager.cc:1397] T c101c1bc9aa94545b65519c2aba9e6b9 P 4dddd225a8d649fabae7c12fa9fe99cc: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260924 11:34:46.162483 15196 raft_consensus.cc:348] T c101c1bc9aa94545b65519c2aba9e6b9 P 4dddd225a8d649fabae7c12fa9fe99cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } }
I20260924 11:34:46.162712 15196 raft_consensus.cc:374] T c101c1bc9aa94545b65519c2aba9e6b9 P 4dddd225a8d649fabae7c12fa9fe99cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:34:46.162786 15196 raft_consensus.cc:729] T c101c1bc9aa94545b65519c2aba9e6b9 P 4dddd225a8d649fabae7c12fa9fe99cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dddd225a8d649fabae7c12fa9fe99cc, State: Initialized, Role: FOLLOWER
I20260924 11:34:46.163159 15196 consensus_queue.cc:260] T c101c1bc9aa94545b65519c2aba9e6b9 P 4dddd225a8d649fabae7c12fa9fe99cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } }
I20260924 11:34:46.165794 15198 tablet_bootstrap.cc:492] T c101c1bc9aa94545b65519c2aba9e6b9 P 2a86a28876eb407596a0452c86e6dd45: Bootstrap starting.
I20260924 11:34:46.166955 15196 ts_tablet_manager.cc:1428] T c101c1bc9aa94545b65519c2aba9e6b9 P 4dddd225a8d649fabae7c12fa9fe99cc: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260924 11:34:46.167783 15198 tablet_bootstrap.cc:654] T c101c1bc9aa94545b65519c2aba9e6b9 P 2a86a28876eb407596a0452c86e6dd45: Neither blocks nor log segments found. Creating new log.
I20260924 11:34:46.170430 15199 tablet_bootstrap.cc:492] T c101c1bc9aa94545b65519c2aba9e6b9 P 432ac21283b94a269b47bdb8f90040b0: No bootstrap required, opened a new log
I20260924 11:34:46.170626 15199 ts_tablet_manager.cc:1397] T c101c1bc9aa94545b65519c2aba9e6b9 P 432ac21283b94a269b47bdb8f90040b0: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260924 11:34:46.171208 15199 raft_consensus.cc:348] T c101c1bc9aa94545b65519c2aba9e6b9 P 432ac21283b94a269b47bdb8f90040b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } }
I20260924 11:34:46.171478 15199 raft_consensus.cc:374] T c101c1bc9aa94545b65519c2aba9e6b9 P 432ac21283b94a269b47bdb8f90040b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:34:46.171586 15199 raft_consensus.cc:729] T c101c1bc9aa94545b65519c2aba9e6b9 P 432ac21283b94a269b47bdb8f90040b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 432ac21283b94a269b47bdb8f90040b0, State: Initialized, Role: FOLLOWER
I20260924 11:34:46.171826 15199 consensus_queue.cc:260] T c101c1bc9aa94545b65519c2aba9e6b9 P 432ac21283b94a269b47bdb8f90040b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } }
I20260924 11:34:46.172767 15199 ts_tablet_manager.cc:1428] T c101c1bc9aa94545b65519c2aba9e6b9 P 432ac21283b94a269b47bdb8f90040b0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 11:34:46.174876 15198 tablet_bootstrap.cc:492] T c101c1bc9aa94545b65519c2aba9e6b9 P 2a86a28876eb407596a0452c86e6dd45: No bootstrap required, opened a new log
I20260924 11:34:46.175051 15198 ts_tablet_manager.cc:1397] T c101c1bc9aa94545b65519c2aba9e6b9 P 2a86a28876eb407596a0452c86e6dd45: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260924 11:34:46.175964 15198 raft_consensus.cc:348] T c101c1bc9aa94545b65519c2aba9e6b9 P 2a86a28876eb407596a0452c86e6dd45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } }
I20260924 11:34:46.176261 15198 raft_consensus.cc:374] T c101c1bc9aa94545b65519c2aba9e6b9 P 2a86a28876eb407596a0452c86e6dd45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:34:46.176357 15198 raft_consensus.cc:729] T c101c1bc9aa94545b65519c2aba9e6b9 P 2a86a28876eb407596a0452c86e6dd45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a86a28876eb407596a0452c86e6dd45, State: Initialized, Role: FOLLOWER
I20260924 11:34:46.176625 15198 consensus_queue.cc:260] T c101c1bc9aa94545b65519c2aba9e6b9 P 2a86a28876eb407596a0452c86e6dd45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } }
I20260924 11:34:46.177176 15198 ts_tablet_manager.cc:1428] T c101c1bc9aa94545b65519c2aba9e6b9 P 2a86a28876eb407596a0452c86e6dd45: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 11:34:46.185732 15174 raft_consensus.cc:482] T c101c1bc9aa94545b65519c2aba9e6b9 P 2a86a28876eb407596a0452c86e6dd45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:34:46.185995 15174 raft_consensus.cc:504] T c101c1bc9aa94545b65519c2aba9e6b9 P 2a86a28876eb407596a0452c86e6dd45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } }
I20260924 11:34:46.186681 15174 leader_election.cc:288] T c101c1bc9aa94545b65519c2aba9e6b9 P 2a86a28876eb407596a0452c86e6dd45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 432ac21283b94a269b47bdb8f90040b0 (127.6.212.2:37311), 4dddd225a8d649fabae7c12fa9fe99cc (127.6.212.1:42819)
I20260924 11:34:46.187515 14757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c101c1bc9aa94545b65519c2aba9e6b9" candidate_uuid: "2a86a28876eb407596a0452c86e6dd45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" is_pre_election: true
I20260924 11:34:46.187781 14757 raft_consensus.cc:2454] T c101c1bc9aa94545b65519c2aba9e6b9 P 4dddd225a8d649fabae7c12fa9fe99cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a86a28876eb407596a0452c86e6dd45 in term 0.
I20260924 11:34:46.188045 14896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c101c1bc9aa94545b65519c2aba9e6b9" candidate_uuid: "2a86a28876eb407596a0452c86e6dd45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "432ac21283b94a269b47bdb8f90040b0" is_pre_election: true
I20260924 11:34:46.188297 14966 leader_election.cc:302] T c101c1bc9aa94545b65519c2aba9e6b9 P 2a86a28876eb407596a0452c86e6dd45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a86a28876eb407596a0452c86e6dd45, 4dddd225a8d649fabae7c12fa9fe99cc; no voters:
I20260924 11:34:46.188334 14896 raft_consensus.cc:2454] T c101c1bc9aa94545b65519c2aba9e6b9 P 432ac21283b94a269b47bdb8f90040b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a86a28876eb407596a0452c86e6dd45 in term 0.
I20260924 11:34:46.188531 15174 raft_consensus.cc:2781] T c101c1bc9aa94545b65519c2aba9e6b9 P 2a86a28876eb407596a0452c86e6dd45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:34:46.188835 15174 raft_consensus.cc:482] T c101c1bc9aa94545b65519c2aba9e6b9 P 2a86a28876eb407596a0452c86e6dd45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:34:46.188993 15174 raft_consensus.cc:3037] T c101c1bc9aa94545b65519c2aba9e6b9 P 2a86a28876eb407596a0452c86e6dd45 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:34:46.191280 15174 raft_consensus.cc:504] T c101c1bc9aa94545b65519c2aba9e6b9 P 2a86a28876eb407596a0452c86e6dd45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } }
I20260924 11:34:46.192200 15174 leader_election.cc:288] T c101c1bc9aa94545b65519c2aba9e6b9 P 2a86a28876eb407596a0452c86e6dd45 [CANDIDATE]: Term 1 election: Requested vote from peers 432ac21283b94a269b47bdb8f90040b0 (127.6.212.2:37311), 4dddd225a8d649fabae7c12fa9fe99cc (127.6.212.1:42819)
I20260924 11:34:46.192698 14757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c101c1bc9aa94545b65519c2aba9e6b9" candidate_uuid: "2a86a28876eb407596a0452c86e6dd45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dddd225a8d649fabae7c12fa9fe99cc"
I20260924 11:34:46.192921 14757 raft_consensus.cc:3037] T c101c1bc9aa94545b65519c2aba9e6b9 P 4dddd225a8d649fabae7c12fa9fe99cc [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:34:46.192984 14896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c101c1bc9aa94545b65519c2aba9e6b9" candidate_uuid: "2a86a28876eb407596a0452c86e6dd45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "432ac21283b94a269b47bdb8f90040b0"
I20260924 11:34:46.193146 14896 raft_consensus.cc:3037] T c101c1bc9aa94545b65519c2aba9e6b9 P 432ac21283b94a269b47bdb8f90040b0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:34:46.195086 14896 raft_consensus.cc:2454] T c101c1bc9aa94545b65519c2aba9e6b9 P 432ac21283b94a269b47bdb8f90040b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a86a28876eb407596a0452c86e6dd45 in term 1.
I20260924 11:34:46.195487 14969 leader_election.cc:302] T c101c1bc9aa94545b65519c2aba9e6b9 P 2a86a28876eb407596a0452c86e6dd45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a86a28876eb407596a0452c86e6dd45, 432ac21283b94a269b47bdb8f90040b0; no voters:
I20260924 11:34:46.195835 15174 raft_consensus.cc:2781] T c101c1bc9aa94545b65519c2aba9e6b9 P 2a86a28876eb407596a0452c86e6dd45 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:34:46.195854 14757 raft_consensus.cc:2454] T c101c1bc9aa94545b65519c2aba9e6b9 P 4dddd225a8d649fabae7c12fa9fe99cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a86a28876eb407596a0452c86e6dd45 in term 1.
I20260924 11:34:46.196269 15174 raft_consensus.cc:686] T c101c1bc9aa94545b65519c2aba9e6b9 P 2a86a28876eb407596a0452c86e6dd45 [term 1 LEADER]: Becoming Leader. State: Replica: 2a86a28876eb407596a0452c86e6dd45, State: Running, Role: LEADER
I20260924 11:34:46.196713 15174 consensus_queue.cc:237] T c101c1bc9aa94545b65519c2aba9e6b9 P 2a86a28876eb407596a0452c86e6dd45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } } peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } }
I20260924 11:34:46.199029 14598 catalog_manager.cc:5495] T c101c1bc9aa94545b65519c2aba9e6b9 P 2a86a28876eb407596a0452c86e6dd45 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a86a28876eb407596a0452c86e6dd45 (127.6.212.3). New cstate: current_term: 1 leader_uuid: "2a86a28876eb407596a0452c86e6dd45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a86a28876eb407596a0452c86e6dd45" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41957 } health_report { overall_health: HEALTHY } } }
I20260924 11:34:46.242272 15215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:46.228246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50402 (local address 127.6.212.3:41957)
0924 11:34:46.237128 (+ 8882us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:46.237138 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:46.237163 (+ 25us) server_negotiation.cc:407] Connection header received
0924 11:34:46.237251 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:46.237263 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:46.237377 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:46.237511 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:34:46.238042 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:46.238938 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:46.240372 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:46.240816 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:46.240965 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:46.241379 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:34:46.241486 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:34:46.241600 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:46.241774 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 11:34:46.241848 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8703,"thread_start_us":8473,"threads_started":1}
I20260924 11:34:46.252370 14757 raft_consensus.cc:1261] T c101c1bc9aa94545b65519c2aba9e6b9 P 4dddd225a8d649fabae7c12fa9fe99cc [term 1 FOLLOWER]: Refusing update from remote peer 2a86a28876eb407596a0452c86e6dd45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:34:46.252424 14896 raft_consensus.cc:1261] T c101c1bc9aa94545b65519c2aba9e6b9 P 432ac21283b94a269b47bdb8f90040b0 [term 1 FOLLOWER]: Refusing update from remote peer 2a86a28876eb407596a0452c86e6dd45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:34:46.252971 15211 consensus_queue.cc:1035] T c101c1bc9aa94545b65519c2aba9e6b9 P 2a86a28876eb407596a0452c86e6dd45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "432ac21283b94a269b47bdb8f90040b0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:34:46.253386 15174 consensus_queue.cc:1035] T c101c1bc9aa94545b65519c2aba9e6b9 P 2a86a28876eb407596a0452c86e6dd45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4dddd225a8d649fabae7c12fa9fe99cc" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:34:46.304284 14598 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:34:46.304539 14598 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:34:46.307163 14598 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c9bf26035dc4b5e9edde52527a113e7: Sending DeleteTablet for 3 replicas of tablet c101c1bc9aa94545b65519c2aba9e6b9
I20260924 11:34:46.308022 14737 tablet_service.cc:1514] Processing DeleteTablet for tablet c101c1bc9aa94545b65519c2aba9e6b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:34:46 UTC) from {username='oryx', principal='oryx/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:52550
I20260924 11:34:46.308037 15015 tablet_service.cc:1514] Processing DeleteTablet for tablet c101c1bc9aa94545b65519c2aba9e6b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:34:46 UTC) from {username='oryx', principal='oryx/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:50376
I20260924 11:34:46.308056 14876 tablet_service.cc:1514] Processing DeleteTablet for tablet c101c1bc9aa94545b65519c2aba9e6b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:34:46 UTC) from {username='oryx', principal='oryx/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:56300
I20260924 11:34:46.308295 15182 tablet_replica.cc:330] T c101c1bc9aa94545b65519c2aba9e6b9 P 4dddd225a8d649fabae7c12fa9fe99cc: stopping tablet replica
I20260924 11:34:46.308317 15180 tablet_replica.cc:330] T c101c1bc9aa94545b65519c2aba9e6b9 P 2a86a28876eb407596a0452c86e6dd45: stopping tablet replica
I20260924 11:34:46.308434 15182 raft_consensus.cc:2229] T c101c1bc9aa94545b65519c2aba9e6b9 P 4dddd225a8d649fabae7c12fa9fe99cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:34:46.308471 15180 raft_consensus.cc:2229] T c101c1bc9aa94545b65519c2aba9e6b9 P 2a86a28876eb407596a0452c86e6dd45 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:34:46.308583 15182 raft_consensus.cc:2258] T c101c1bc9aa94545b65519c2aba9e6b9 P 4dddd225a8d649fabae7c12fa9fe99cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:34:46.308732 15180 raft_consensus.cc:2258] T c101c1bc9aa94545b65519c2aba9e6b9 P 2a86a28876eb407596a0452c86e6dd45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:34:46.309018 15181 tablet_replica.cc:330] T c101c1bc9aa94545b65519c2aba9e6b9 P 432ac21283b94a269b47bdb8f90040b0: stopping tablet replica
I20260924 11:34:46.309192 15181 raft_consensus.cc:2229] T c101c1bc9aa94545b65519c2aba9e6b9 P 432ac21283b94a269b47bdb8f90040b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:34:46.309386 15181 raft_consensus.cc:2258] T c101c1bc9aa94545b65519c2aba9e6b9 P 432ac21283b94a269b47bdb8f90040b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:34:46.309530 15182 ts_tablet_manager.cc:1905] T c101c1bc9aa94545b65519c2aba9e6b9 P 4dddd225a8d649fabae7c12fa9fe99cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:34:46.310413 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 14671
I20260924 11:34:46.310552 15181 ts_tablet_manager.cc:1905] T c101c1bc9aa94545b65519c2aba9e6b9 P 432ac21283b94a269b47bdb8f90040b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:34:46.310758 15180 ts_tablet_manager.cc:1905] T c101c1bc9aa94545b65519c2aba9e6b9 P 2a86a28876eb407596a0452c86e6dd45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:34:46.316061 15182 ts_tablet_manager.cc:1918] T c101c1bc9aa94545b65519c2aba9e6b9 P 4dddd225a8d649fabae7c12fa9fe99cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 11:34:46.316227 15182 log.cc:1196] T c101c1bc9aa94545b65519c2aba9e6b9 P 4dddd225a8d649fabae7c12fa9fe99cc: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-0/wal/wals/c101c1bc9aa94545b65519c2aba9e6b9
I20260924 11:34:46.316636 15182 ts_tablet_manager.cc:1939] T c101c1bc9aa94545b65519c2aba9e6b9 P 4dddd225a8d649fabae7c12fa9fe99cc: Deleting consensus metadata
I20260924 11:34:46.317222 15180 ts_tablet_manager.cc:1918] T c101c1bc9aa94545b65519c2aba9e6b9 P 2a86a28876eb407596a0452c86e6dd45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 11:34:46.317384 15180 log.cc:1196] T c101c1bc9aa94545b65519c2aba9e6b9 P 2a86a28876eb407596a0452c86e6dd45: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-2/wal/wals/c101c1bc9aa94545b65519c2aba9e6b9
I20260924 11:34:46.317704 14584 catalog_manager.cc:4841] TS 4dddd225a8d649fabae7c12fa9fe99cc (127.6.212.1:42819): tablet c101c1bc9aa94545b65519c2aba9e6b9 (table test-table [id=9f9439de7a19460a982e5cb6390d26ba]) successfully deleted
I20260924 11:34:46.317991 15180 ts_tablet_manager.cc:1939] T c101c1bc9aa94545b65519c2aba9e6b9 P 2a86a28876eb407596a0452c86e6dd45: Deleting consensus metadata
I20260924 11:34:46.318336 15181 ts_tablet_manager.cc:1918] T c101c1bc9aa94545b65519c2aba9e6b9 P 432ac21283b94a269b47bdb8f90040b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 11:34:46.318524 15181 log.cc:1196] T c101c1bc9aa94545b65519c2aba9e6b9 P 432ac21283b94a269b47bdb8f90040b0: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790249617057452-6992-0/minicluster-data/ts-1/wal/wals/c101c1bc9aa94545b65519c2aba9e6b9
I20260924 11:34:46.318969 14586 catalog_manager.cc:4841] TS 2a86a28876eb407596a0452c86e6dd45 (127.6.212.3:41957): tablet c101c1bc9aa94545b65519c2aba9e6b9 (table test-table [id=9f9439de7a19460a982e5cb6390d26ba]) successfully deleted
I20260924 11:34:46.319177 15181 ts_tablet_manager.cc:1939] T c101c1bc9aa94545b65519c2aba9e6b9 P 432ac21283b94a269b47bdb8f90040b0: Deleting consensus metadata
I20260924 11:34:46.320226 14587 catalog_manager.cc:4841] TS 432ac21283b94a269b47bdb8f90040b0 (127.6.212.2:37311): tablet c101c1bc9aa94545b65519c2aba9e6b9 (table test-table [id=9f9439de7a19460a982e5cb6390d26ba]) successfully deleted
Sep 24 11:34:46 dist-test-slave-cq7c krb5kdc[14555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.212.1@KRBTEST.COM
I20260924 11:34:46.332651 15205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:46.312006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47412 (local address 127.6.212.1:42819)
0924 11:34:46.312279 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:46.312286 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:46.319699 (+ 7413us) server_negotiation.cc:407] Connection header received
0924 11:34:46.319830 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:46.319836 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:46.319917 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:46.320030 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:46.321254 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:46.321950 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:46.323114 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:46.323589 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:46.323718 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:46.327334 (+ 3616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:46.327347 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:46.327408 (+ 61us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:46.327458 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:46.330042 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:46.330672 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:46.330683 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:46.330691 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:46.330810 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:46.331161 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:46.331167 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:46.331171 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:46.331576 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:46.331861 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:46.332148 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 11:34:46.332354 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
I20260924 11:34:46.333250 14802 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:34:46.458262 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 14671
I20260924 11:34:46.480809 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 14811
Sep 24 11:34:46 dist-test-slave-cq7c krb5kdc[14555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.212.2@KRBTEST.COM
I20260924 11:34:46.495142 15223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:46.481546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47252 (local address 127.6.212.2:37311)
0924 11:34:46.481981 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:46.481986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:46.482002 (+ 16us) server_negotiation.cc:407] Connection header received
0924 11:34:46.482054 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:46.482057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:46.482117 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:46.482206 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:46.483506 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:46.484291 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:46.485537 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:46.485913 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:46.486007 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:46.489932 (+ 3925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:46.489945 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:46.489961 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:46.489994 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:46.492884 (+ 2890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:46.493533 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:46.493540 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:46.493542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:46.493658 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:46.493953 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:46.493956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:46.493958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:46.494270 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:46.494500 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:46.494702 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 11:34:46.494864 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":225,"threads_started":1}
I20260924 11:34:46.495828 14937 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:34:46.620239 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 14811
I20260924 11:34:46.642189 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 14950
Sep 24 11:34:46 dist-test-slave-cq7c krb5kdc[14555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.212.3@KRBTEST.COM
I20260924 11:34:46.657467 15215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:46.642981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54722 (local address 127.6.212.3:41957)
0924 11:34:46.643153 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:46.643157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:46.643174 (+ 17us) server_negotiation.cc:407] Connection header received
0924 11:34:46.643396 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:46.643405 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:46.643483 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:46.643571 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:46.644761 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:46.645474 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:46.646663 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:46.647087 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:46.647182 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:46.651424 (+ 4242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:46.651440 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:46.651455 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:46.651497 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:46.654557 (+ 3060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:46.655505 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:46.655517 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:46.655524 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:46.655715 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:46.656090 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:46.656094 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:46.656096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:46.656427 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:46.656642 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:46.656939 (+ 297us) server_negotiation.cc:299] Negotiation successful
0924 11:34:46.657126 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 11:34:46.658099 15080 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:34:46.782553 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 14950
I20260924 11:34:46.804584 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 14568
I20260924 11:34:46.805717 14640 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:34:46.921041 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 14568
2026-09-24T11:34:46Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5631 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790249617057452-6992-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:34:47 dist-test-slave-cq7c krb5kdc[15229](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:34:47 dist-test-slave-cq7c krb5kdc[15229](info): set up 2 sockets
Sep 24 11:34:47 dist-test-slave-cq7c krb5kdc[15229](info): commencing operation
krb5kdc: starting...
W20260924 11:34:48.040347 6992 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Sep 24 11:34:48 dist-test-slave-cq7c krb5kdc[15229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:34:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:34:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.212.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
I20260924 11:34:48.271350 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:33955
--webserver_interface=127.6.212.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:35709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:33955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:34:48.604756 15242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:48.605046 15242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:48.605134 15242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:48.612413 15242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:34:48.612519 15242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:48.612618 15242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:48.612715 15242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:34:48.612816 15242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:34:48.622927 15242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:35709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:33955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:33955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.212.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:34:48.624348 15242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:48.625759 15242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:48.638190 15247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:48.638257 15248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:48.638288 15250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:48.639233 15242 server_base.cc:1034] running on GCE node
I20260924 11:34:48.640100 15242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:48.641507 15242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:48.642704 15242 hybrid_clock.cc:648] HybridClock initialized: now 1790249688642609 us; error 88 us; skew 500 ppm
I20260924 11:34:48.643071 15242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T11:34:48Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1787 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:34:48 dist-test-slave-cq7c krb5kdc[15256](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:34:48 dist-test-slave-cq7c krb5kdc[15256](info): set up 2 sockets
Sep 24 11:34:48 dist-test-slave-cq7c krb5kdc[15256](info): commencing operation
krb5kdc: starting...
W20260924 11:34:49.841060 6992 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Sep 24 11:34:49 dist-test-slave-cq7c krb5kdc[15256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:34:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:34:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.212.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
I20260924 11:34:50.081328 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:35549
--webserver_interface=127.6.212.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:40115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:35549
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:34:50.405233 15269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:50.405582 15269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:50.405754 15269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:50.412865 15269 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:34:50.413012 15269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:50.413121 15269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:50.413231 15269 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:34:50.413329 15269 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:34:50.422981 15269 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:40115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:35549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:35549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.212.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:34:50.424346 15269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:50.425660 15269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:50.437487 15274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:50.437525 15277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:50.437599 15275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:50.438723 15269 server_base.cc:1034] running on GCE node
I20260924 11:34:50.439585 15269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:50.440984 15269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:50.442215 15269 hybrid_clock.cc:648] HybridClock initialized: now 1790249690442169 us; error 39 us; skew 500 ppm
I20260924 11:34:50.442550 15269 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:34:50 dist-test-slave-cq7c krb5kdc[15256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249690, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:50.450042 15269 init.cc:377] Logged in from keytab as kudu/127.6.212.62@KRBTEST.COM (short username kudu)
I20260924 11:34:50.451624 15269 webserver.cc:460] Webserver started at http://127.6.212.62:45915/ using document root <none> and password file <none>
I20260924 11:34:50.452329 15269 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:50.452440 15269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:50.452683 15269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:34:50.455621 15269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "824c3ebb436840b79634de262ecfde40"
format_stamp: "Formatted at 2026-09-24 11:34:50 on dist-test-slave-cq7c"
server_key: "63fa24e1fa955ff787f2d5d5a5f80158"
server_key_iv: "7c1d8fe3a06c0565f1defc2de2abe0dc"
server_key_version: "encryptionkey@0"
I20260924 11:34:50.456341 15269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "824c3ebb436840b79634de262ecfde40"
format_stamp: "Formatted at 2026-09-24 11:34:50 on dist-test-slave-cq7c"
server_key: "63fa24e1fa955ff787f2d5d5a5f80158"
server_key_iv: "7c1d8fe3a06c0565f1defc2de2abe0dc"
server_key_version: "encryptionkey@0"
I20260924 11:34:50.461329 15269 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 11:34:50.465121 15284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:50.466492 15269 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:34:50.466662 15269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "824c3ebb436840b79634de262ecfde40"
format_stamp: "Formatted at 2026-09-24 11:34:50 on dist-test-slave-cq7c"
server_key: "63fa24e1fa955ff787f2d5d5a5f80158"
server_key_iv: "7c1d8fe3a06c0565f1defc2de2abe0dc"
server_key_version: "encryptionkey@0"
I20260924 11:34:50.466843 15269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:50.481411 15269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:50.485512 15269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:34:50.485798 15269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:50.504520 15269 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.62:35549
I20260924 11:34:50.504557 15335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.62:35549 every 8 connection(s)
I20260924 11:34:50.505926 15269 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:34:50.509789 15336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:34:50.513582 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 15269
I20260924 11:34:50.513815 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:34:50.514441 6992 external_mini_cluster.cc:1428] Setting key 49d00ecbd0bf75ddadd8ffff8fd22b72
I20260924 11:34:50.522360 15336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 824c3ebb436840b79634de262ecfde40: Bootstrap starting.
I20260924 11:34:50.526434 15336 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 824c3ebb436840b79634de262ecfde40: Neither blocks nor log segments found. Creating new log.
I20260924 11:34:50.527886 15336 log.cc:824] T 00000000000000000000000000000000 P 824c3ebb436840b79634de262ecfde40: Log is configured to *not* fsync() on all Append() calls
I20260924 11:34:50.531278 15336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 824c3ebb436840b79634de262ecfde40: No bootstrap required, opened a new log
Sep 24 11:34:50 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:34:50.542073 15339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:50.516451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59120 (local address 127.6.212.62:35549)
0924 11:34:50.517235 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:50.517253 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:50.517301 (+ 48us) server_negotiation.cc:407] Connection header received
0924 11:34:50.520362 (+ 3061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:50.520402 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:50.520940 (+ 538us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:50.521443 (+ 503us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:50.522757 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:50.525046 (+ 2289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:50.526320 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:50.527530 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:50.527749 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:50.532463 (+ 4714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:50.532475 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:50.532505 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:50.532551 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:50.536979 (+ 4428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:50.537728 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:50.537739 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:50.537749 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:50.537997 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:50.538363 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:50.538368 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:50.538372 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:50.539209 (+ 837us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:50.539583 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:50.539927 (+ 344us) server_negotiation.cc:299] Negotiation successful
0924 11:34:50.540454 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":317,"threads_started":1}
I20260924 11:34:50.543339 15336 raft_consensus.cc:348] T 00000000000000000000000000000000 P 824c3ebb436840b79634de262ecfde40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824c3ebb436840b79634de262ecfde40" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 35549 } }
I20260924 11:34:50.543810 15336 raft_consensus.cc:374] T 00000000000000000000000000000000 P 824c3ebb436840b79634de262ecfde40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:34:50.543942 15336 raft_consensus.cc:729] T 00000000000000000000000000000000 P 824c3ebb436840b79634de262ecfde40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 824c3ebb436840b79634de262ecfde40, State: Initialized, Role: FOLLOWER
I20260924 11:34:50.544574 15336 consensus_queue.cc:260] T 00000000000000000000000000000000 P 824c3ebb436840b79634de262ecfde40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824c3ebb436840b79634de262ecfde40" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 35549 } }
I20260924 11:34:50.544834 15336 raft_consensus.cc:388] T 00000000000000000000000000000000 P 824c3ebb436840b79634de262ecfde40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:34:50.545058 15336 raft_consensus.cc:482] T 00000000000000000000000000000000 P 824c3ebb436840b79634de262ecfde40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:34:50.545275 15336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 824c3ebb436840b79634de262ecfde40 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:34:50.547802 15336 raft_consensus.cc:504] T 00000000000000000000000000000000 P 824c3ebb436840b79634de262ecfde40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824c3ebb436840b79634de262ecfde40" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 35549 } }
I20260924 11:34:50.548260 15336 leader_election.cc:302] T 00000000000000000000000000000000 P 824c3ebb436840b79634de262ecfde40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 824c3ebb436840b79634de262ecfde40; no voters:
I20260924 11:34:50.549012 15336 leader_election.cc:288] T 00000000000000000000000000000000 P 824c3ebb436840b79634de262ecfde40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:34:50.549110 15341 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 824c3ebb436840b79634de262ecfde40 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:34:50.550060 15341 raft_consensus.cc:686] T 00000000000000000000000000000000 P 824c3ebb436840b79634de262ecfde40 [term 1 LEADER]: Becoming Leader. State: Replica: 824c3ebb436840b79634de262ecfde40, State: Running, Role: LEADER
I20260924 11:34:50.550702 15341 consensus_queue.cc:237] T 00000000000000000000000000000000 P 824c3ebb436840b79634de262ecfde40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824c3ebb436840b79634de262ecfde40" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 35549 } }
I20260924 11:34:50.551672 15336 sys_catalog.cc:564] T 00000000000000000000000000000000 P 824c3ebb436840b79634de262ecfde40 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:34:50.553491 15343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 824c3ebb436840b79634de262ecfde40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "824c3ebb436840b79634de262ecfde40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824c3ebb436840b79634de262ecfde40" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 35549 } } }
I20260924 11:34:50.554364 15343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 824c3ebb436840b79634de262ecfde40 [sys.catalog]: This master's current role is: LEADER
I20260924 11:34:50.554999 15342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 824c3ebb436840b79634de262ecfde40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 824c3ebb436840b79634de262ecfde40. Latest consensus state: current_term: 1 leader_uuid: "824c3ebb436840b79634de262ecfde40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824c3ebb436840b79634de262ecfde40" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 35549 } } }
I20260924 11:34:50.555209 15349 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:34:50.555321 15342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 824c3ebb436840b79634de262ecfde40 [sys.catalog]: This master's current role is: LEADER
I20260924 11:34:50.562968 15349 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:34:50.573765 15349 catalog_manager.cc:1347] Generated new cluster ID: 443e8a6bb26c4a7ba34d29a2213decd3
I20260924 11:34:50.573905 15349 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:34:50.584935 15349 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:34:50.586818 15349 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:34:50.605582 15349 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 824c3ebb436840b79634de262ecfde40: Generated new TSK 0
I20260924 11:34:50.606786 15349 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.212.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab.
Entry for principal kudu/127.6.212.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab.
I20260924 11:34:50.672322 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.1:0
--local_ip_for_outbound_sockets=127.6.212.1
--webserver_interface=127.6.212.1
--webserver_port=0
--tserver_master_addrs=127.6.212.62:35549
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab
--principal=kudu/127.6.212.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:40115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:34:51.010522 15362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:51.010843 15362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:51.011018 15362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:51.018097 15362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:51.018246 15362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:51.018440 15362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.1
I20260924 11:34:51.028349 15362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:40115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab
--principal=kudu/127.6.212.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.212.1
--webserver_port=0
--tserver_master_addrs=127.6.212.62:35549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.1
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:34:51.029712 15362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:51.031049 15362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:51.045292 15367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:51.045356 15370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:51.045324 15368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:51.046289 15362 server_base.cc:1034] running on GCE node
I20260924 11:34:51.046996 15362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:51.048053 15362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:51.049299 15362 hybrid_clock.cc:648] HybridClock initialized: now 1790249691049244 us; error 49 us; skew 500 ppm
I20260924 11:34:51.049640 15362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:34:51 dist-test-slave-cq7c krb5kdc[15256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249691, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:51.053102 15362 init.cc:377] Logged in from keytab as kudu/127.6.212.1@KRBTEST.COM (short username kudu)
I20260924 11:34:51.054713 15362 webserver.cc:460] Webserver started at http://127.6.212.1:43897/ using document root <none> and password file <none>
I20260924 11:34:51.055477 15362 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:51.055624 15362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:51.055958 15362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:34:51.058928 15362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-0/data/instance:
uuid: "ff693c2e8abc426bb2d9b3fc69c71f3c"
format_stamp: "Formatted at 2026-09-24 11:34:51 on dist-test-slave-cq7c"
server_key: "b59b9d50f3318e7f4a164362775e0e1b"
server_key_iv: "32fc971e08724de783317c44ec983af4"
server_key_version: "encryptionkey@0"
I20260924 11:34:51.059749 15362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance:
uuid: "ff693c2e8abc426bb2d9b3fc69c71f3c"
format_stamp: "Formatted at 2026-09-24 11:34:51 on dist-test-slave-cq7c"
server_key: "b59b9d50f3318e7f4a164362775e0e1b"
server_key_iv: "32fc971e08724de783317c44ec983af4"
server_key_version: "encryptionkey@0"
I20260924 11:34:51.064931 15362 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260924 11:34:51.068665 15377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:51.070286 15362 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260924 11:34:51.070487 15362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-0/wal
uuid: "ff693c2e8abc426bb2d9b3fc69c71f3c"
format_stamp: "Formatted at 2026-09-24 11:34:51 on dist-test-slave-cq7c"
server_key: "b59b9d50f3318e7f4a164362775e0e1b"
server_key_iv: "32fc971e08724de783317c44ec983af4"
server_key_version: "encryptionkey@0"
I20260924 11:34:51.070708 15362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:51.092404 15362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:51.096807 15362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:34:51.097121 15362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:51.098145 15362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:34:51.099524 15362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:34:51.099627 15362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:51.099795 15362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:34:51.099869 15362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:51.133080 15362 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.1:45801
I20260924 11:34:51.133122 15489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.1:45801 every 8 connection(s)
I20260924 11:34:51.134428 15362 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
I20260924 11:34:51.137645 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 15362
I20260924 11:34:51.137959 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance
I20260924 11:34:51.138820 6992 external_mini_cluster.cc:1428] Setting key 9fb1b77ad91ba455603c69485d742431
Sep 24 11:34:51 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249691, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:34:51.159432 15492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:51.137178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:34249 (local address 127.6.212.62:35549)
0924 11:34:51.137735 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:51.137742 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:51.138921 (+ 1179us) server_negotiation.cc:407] Connection header received
0924 11:34:51.141515 (+ 2594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:51.141519 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:51.141601 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:51.141687 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:51.144266 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:51.145033 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:51.147676 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:51.148083 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:51.148185 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:51.152621 (+ 4436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:51.152627 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:51.152631 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:51.152667 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:51.155702 (+ 3035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:51.156725 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:51.156729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:51.156732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:51.156902 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:51.157484 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:51.157489 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:51.157493 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:51.157804 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:51.157928 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:51.158948 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0924 11:34:51.159116 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":279,"threads_started":1}
I20260924 11:34:51.161255 15493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:51.137766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.1:34249)
0924 11:34:51.138719 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:51.138780 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:51.141144 (+ 2364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:51.141997 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:51.142012 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:51.142910 (+ 898us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:51.144050 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:51.144083 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:51.145205 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:51.145213 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:51.147522 (+ 2309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:51.147541 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:51.148336 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:51.148346 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:51.148703 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:51.149524 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:51.149572 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:51.152259 (+ 2687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:51.155871 (+ 3612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:51.155883 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:51.155905 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:51.156592 (+ 687us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:51.157058 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:51.157066 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:51.157072 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:51.157341 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:51.157951 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:51.157962 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:51.158494 (+ 532us) client_negotiation.cc:769] Sending connection context
0924 11:34:51.158858 (+ 364us) client_negotiation.cc:240] Negotiation successful
0924 11:34:51.159524 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":359,"threads_started":1}
I20260924 11:34:51.163043 15490 heartbeater.cc:347] Connected to a master server at 127.6.212.62:35549
I20260924 11:34:51.163405 15490 heartbeater.cc:464] Registering TS with master...
I20260924 11:34:51.164152 15490 heartbeater.cc:511] Master 127.6.212.62:35549 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.212.2@KRBTEST.COM; defaulting to no policy
I20260924 11:34:51.166498 15301 ts_manager.cc:194] Registered new tserver with Master: ff693c2e8abc426bb2d9b3fc69c71f3c (127.6.212.1:45801)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.2@KRBTEST.COM" created.
I20260924 11:34:51.168130 15301 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.1@KRBTEST.COM'} at 127.6.212.1:34249
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab.
Entry for principal kudu/127.6.212.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab.
I20260924 11:34:51.195741 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.2:0
--local_ip_for_outbound_sockets=127.6.212.2
--webserver_interface=127.6.212.2
--webserver_port=0
--tserver_master_addrs=127.6.212.62:35549
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab
--principal=kudu/127.6.212.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:40115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:34:51.529610 15496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:51.529896 15496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:51.530046 15496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:51.537465 15496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:51.537585 15496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:51.537755 15496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.2
I20260924 11:34:51.547700 15496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:40115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab
--principal=kudu/127.6.212.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.212.2
--webserver_port=0
--tserver_master_addrs=127.6.212.62:35549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.2
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:34:51.549080 15496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:51.550491 15496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:51.563828 15502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:51.563993 15504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:51.564106 15501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:51.565097 15496 server_base.cc:1034] running on GCE node
I20260924 11:34:51.565763 15496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:51.566833 15496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:51.568043 15496 hybrid_clock.cc:648] HybridClock initialized: now 1790249691567986 us; error 61 us; skew 500 ppm
I20260924 11:34:51.568391 15496 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:34:51 dist-test-slave-cq7c krb5kdc[15256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249691, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:51.571630 15496 init.cc:377] Logged in from keytab as kudu/127.6.212.2@KRBTEST.COM (short username kudu)
I20260924 11:34:51.573141 15496 webserver.cc:460] Webserver started at http://127.6.212.2:35139/ using document root <none> and password file <none>
I20260924 11:34:51.573885 15496 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:51.574033 15496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:51.574347 15496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:34:51.577420 15496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-1/data/instance:
uuid: "55d48eafb42543b78206c7a1b4c69f78"
format_stamp: "Formatted at 2026-09-24 11:34:51 on dist-test-slave-cq7c"
server_key: "403bcb3e0746a284724790bd98a97a12"
server_key_iv: "7f39f1d21cce3902e0ef5d5887807017"
server_key_version: "encryptionkey@0"
I20260924 11:34:51.578178 15496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance:
uuid: "55d48eafb42543b78206c7a1b4c69f78"
format_stamp: "Formatted at 2026-09-24 11:34:51 on dist-test-slave-cq7c"
server_key: "403bcb3e0746a284724790bd98a97a12"
server_key_iv: "7f39f1d21cce3902e0ef5d5887807017"
server_key_version: "encryptionkey@0"
I20260924 11:34:51.583439 15496 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 11:34:51.586985 15511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:51.588531 15496 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 11:34:51.588709 15496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-1/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-1/wal
uuid: "55d48eafb42543b78206c7a1b4c69f78"
format_stamp: "Formatted at 2026-09-24 11:34:51 on dist-test-slave-cq7c"
server_key: "403bcb3e0746a284724790bd98a97a12"
server_key_iv: "7f39f1d21cce3902e0ef5d5887807017"
server_key_version: "encryptionkey@0"
I20260924 11:34:51.588927 15496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:51.608129 15496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:51.612542 15496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:34:51.612849 15496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:51.613823 15496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:34:51.615154 15496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:34:51.615258 15496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:51.615434 15496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:34:51.615510 15496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260924 11:34:51.649224 15496 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.2:41307
I20260924 11:34:51.649287 15623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.2:41307 every 8 connection(s)
I20260924 11:34:51.650627 15496 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
I20260924 11:34:51.658946 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 15496
I20260924 11:34:51.659139 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance
I20260924 11:34:51.659638 6992 external_mini_cluster.cc:1428] Setting key 6a11e1142d6c88ae586dba97b2835038
Sep 24 11:34:51 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249691, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.2@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:34:51.672049 15492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:51.653341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:48437 (local address 127.6.212.62:35549)
0924 11:34:51.653562 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:51.653569 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:51.655030 (+ 1461us) server_negotiation.cc:407] Connection header received
0924 11:34:51.656902 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:51.656907 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:51.656974 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:51.657064 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:51.658693 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:51.659619 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:51.661488 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:51.661935 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:51.662029 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:51.666389 (+ 4360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:51.666394 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:51.666398 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:51.666428 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:51.668947 (+ 2519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:51.669739 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:51.669746 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:51.669752 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:51.669886 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:51.670314 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:51.670319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:51.670322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:51.670655 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:51.670801 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:51.671488 (+ 687us) server_negotiation.cc:299] Negotiation successful
0924 11:34:51.671700 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 11:34:51.673090 15626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:51.654023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.2:48437)
0924 11:34:51.654879 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:51.654922 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:51.656647 (+ 1725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:51.657318 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:51.657334 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:51.657916 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:51.658520 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:51.658538 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:51.659771 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:51.659776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:51.661335 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:51.661348 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:51.662193 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:51.662200 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:51.662546 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:51.663313 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:51.663378 (+ 65us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:51.666156 (+ 2778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:51.669089 (+ 2933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:51.669103 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:51.669120 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:51.669623 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:51.670003 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:51.670008 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:51.670012 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:51.670195 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:51.670778 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:51.670786 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:51.671119 (+ 333us) client_negotiation.cc:769] Sending connection context
0924 11:34:51.671416 (+ 297us) client_negotiation.cc:240] Negotiation successful
0924 11:34:51.671961 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":357,"threads_started":1}
I20260924 11:34:51.674512 15624 heartbeater.cc:347] Connected to a master server at 127.6.212.62:35549
I20260924 11:34:51.674844 15624 heartbeater.cc:464] Registering TS with master...
I20260924 11:34:51.675698 15624 heartbeater.cc:511] Master 127.6.212.62:35549 requested a full tablet report, sending...
I20260924 11:34:51.677290 15301 ts_manager.cc:194] Registered new tserver with Master: 55d48eafb42543b78206c7a1b4c69f78 (127.6.212.2:41307)
I20260924 11:34:51.678670 15301 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.2@KRBTEST.COM'} at 127.6.212.2:48437
WARNING: no policy specified for kudu/127.6.212.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab.
Entry for principal kudu/127.6.212.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab.
I20260924 11:34:51.715261 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.3:0
--local_ip_for_outbound_sockets=127.6.212.3
--webserver_interface=127.6.212.3
--webserver_port=0
--tserver_master_addrs=127.6.212.62:35549
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab
--principal=kudu/127.6.212.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:40115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:34:52.049628 15629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:52.049935 15629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:52.050028 15629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:52.057657 15629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:52.057924 15629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:52.058107 15629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.3
I20260924 11:34:52.068290 15629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:40115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab
--principal=kudu/127.6.212.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.212.3
--webserver_port=0
--tserver_master_addrs=127.6.212.62:35549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.3
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:34:52.069664 15629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:52.071130 15629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:52.085850 15635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:52.085850 15637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:52.086241 15634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:52.086615 15629 server_base.cc:1034] running on GCE node
I20260924 11:34:52.087306 15629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:52.088347 15629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:52.089594 15629 hybrid_clock.cc:648] HybridClock initialized: now 1790249692089523 us; error 62 us; skew 500 ppm
I20260924 11:34:52.089939 15629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:34:52 dist-test-slave-cq7c krb5kdc[15256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249692, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:52.093338 15629 init.cc:377] Logged in from keytab as kudu/127.6.212.3@KRBTEST.COM (short username kudu)
I20260924 11:34:52.095031 15629 webserver.cc:460] Webserver started at http://127.6.212.3:36137/ using document root <none> and password file <none>
I20260924 11:34:52.095813 15629 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:52.095961 15629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:52.096278 15629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:34:52.099933 15629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-2/data/instance:
uuid: "5c83d66b3a8f4c2f825acd5dd8afc6dd"
format_stamp: "Formatted at 2026-09-24 11:34:52 on dist-test-slave-cq7c"
server_key: "d04de44921f2e6afb15b2b61cb3c777f"
server_key_iv: "0f5987a15fb1ff88dc55fe93ea56295b"
server_key_version: "encryptionkey@0"
I20260924 11:34:52.100834 15629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance:
uuid: "5c83d66b3a8f4c2f825acd5dd8afc6dd"
format_stamp: "Formatted at 2026-09-24 11:34:52 on dist-test-slave-cq7c"
server_key: "d04de44921f2e6afb15b2b61cb3c777f"
server_key_iv: "0f5987a15fb1ff88dc55fe93ea56295b"
server_key_version: "encryptionkey@0"
I20260924 11:34:52.106673 15629 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 11:34:52.110592 15644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:52.112075 15629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 11:34:52.112275 15629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-2/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-2/wal
uuid: "5c83d66b3a8f4c2f825acd5dd8afc6dd"
format_stamp: "Formatted at 2026-09-24 11:34:52 on dist-test-slave-cq7c"
server_key: "d04de44921f2e6afb15b2b61cb3c777f"
server_key_iv: "0f5987a15fb1ff88dc55fe93ea56295b"
server_key_version: "encryptionkey@0"
I20260924 11:34:52.112500 15629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:52.131177 15629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:52.136121 15629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:34:52.136499 15629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:52.137676 15629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:34:52.139209 15629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:34:52.139325 15629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:52.139555 15629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:34:52.139639 15629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:52.172199 15490 heartbeater.cc:503] Master 127.6.212.62:35549 was elected leader, sending a full tablet report...
I20260924 11:34:52.174559 15629 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.3:37151
I20260924 11:34:52.174603 15756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.3:37151 every 8 connection(s)
I20260924 11:34:52.176019 15629 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
I20260924 11:34:52.178453 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 15629
I20260924 11:34:52.178648 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance
I20260924 11:34:52.179142 6992 external_mini_cluster.cc:1428] Setting key fa67ce630bd8cc859b71014be1165d55
Sep 24 11:34:52 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249692, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.3@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:34:52.196691 15759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:52.178664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:39485 (local address 127.6.212.62:35549)
0924 11:34:52.179150 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:52.179154 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:52.182374 (+ 3220us) server_negotiation.cc:407] Connection header received
0924 11:34:52.182425 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:52.182428 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:52.182498 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:52.182581 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:52.184273 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:52.185083 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:52.186640 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:52.187032 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:52.187133 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:52.190587 (+ 3454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:52.190592 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:52.190595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:52.190628 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:52.193553 (+ 2925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:52.194297 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:52.194303 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:52.194307 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:52.194431 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:52.194867 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:52.194880 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:52.194884 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:52.195140 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:52.195292 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:52.196104 (+ 812us) server_negotiation.cc:299] Negotiation successful
0924 11:34:52.196312 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":265,"threads_started":1}
I20260924 11:34:52.197649 15760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:52.179406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.3:39485)
0924 11:34:52.180328 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:52.180373 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:52.182085 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:52.182850 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:52.182860 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:52.183496 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:52.184107 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:52.184126 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:52.185222 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:52.185226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:52.186502 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:52.186514 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:52.187231 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:52.187235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:52.187527 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:52.188168 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:52.188195 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:52.190383 (+ 2188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:34:52.193705 (+ 3322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:52.193714 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:52.193733 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:52.194165 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:52.194541 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:34:52.194546 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:34:52.194550 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:34:52.194748 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:34:52.195270 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:34:52.195278 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:34:52.195709 (+ 431us) client_negotiation.cc:769] Sending connection context
0924 11:34:52.196014 (+ 305us) client_negotiation.cc:240] Negotiation successful
0924 11:34:52.196566 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":399,"threads_started":1}
I20260924 11:34:52.199055 15757 heartbeater.cc:347] Connected to a master server at 127.6.212.62:35549
I20260924 11:34:52.199420 15757 heartbeater.cc:464] Registering TS with master...
I20260924 11:34:52.200130 15757 heartbeater.cc:511] Master 127.6.212.62:35549 requested a full tablet report, sending...
I20260924 11:34:52.201502 15301 ts_manager.cc:194] Registered new tserver with Master: 5c83d66b3a8f4c2f825acd5dd8afc6dd (127.6.212.3:37151)
I20260924 11:34:52.202299 15301 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.3@KRBTEST.COM'} at 127.6.212.3:39485
I20260924 11:34:52.206684 6992 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.6.212.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.212.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.212.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.1.keytab.
Entry for principal oryx/127.6.212.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.1.keytab.
WARNING: no policy specified for oryx/127.6.212.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.212.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.212.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.2.keytab.
Entry for principal oryx/127.6.212.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.2.keytab.
WARNING: no policy specified for oryx/127.6.212.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.212.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.212.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.3.keytab.
Entry for principal oryx/127.6.212.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.3.keytab.
I20260924 11:34:52.354759 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 15362
Sep 24 11:34:52 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.1@KRBTEST.COM
I20260924 11:34:52.372087 15768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:52.356044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40126 (local address 127.6.212.1:45801)
0924 11:34:52.356754 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:52.356762 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:52.356789 (+ 27us) server_negotiation.cc:407] Connection header received
0924 11:34:52.356853 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:52.356861 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:52.357168 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:52.357320 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:52.358669 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:52.359808 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:52.361377 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:52.362113 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:52.362234 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:52.366350 (+ 4116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:52.366361 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:52.366377 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:52.366411 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:52.369162 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:52.369803 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:52.369815 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:52.369820 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:52.370028 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:52.370471 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:52.370477 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:52.370486 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:52.370808 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:52.371113 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:52.371468 (+ 355us) server_negotiation.cc:299] Negotiation successful
0924 11:34:52.371738 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":392,"threads_started":1}
I20260924 11:34:52.373397 15485 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:34:52.457295 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 15362
I20260924 11:34:52.476171 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 15496
Sep 24 11:34:52 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.2@KRBTEST.COM
I20260924 11:34:52.493979 15770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:52.477199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35244 (local address 127.6.212.2:41307)
0924 11:34:52.477790 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:52.477797 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:52.477824 (+ 27us) server_negotiation.cc:407] Connection header received
0924 11:34:52.477887 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:52.477895 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:52.478106 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:52.478243 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:52.479698 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:52.480761 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:52.482185 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:52.482807 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:52.482959 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:52.487679 (+ 4720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:52.487692 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:52.487707 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:52.487750 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:52.490709 (+ 2959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:52.491709 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:52.491716 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:52.491719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:52.491841 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:52.492308 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:52.492312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:52.492314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:52.492723 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:52.493019 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:52.493431 (+ 412us) server_negotiation.cc:299] Negotiation successful
0924 11:34:52.493684 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":301,"threads_started":1}
I20260924 11:34:52.495224 15619 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:34:52.583817 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 15496
I20260924 11:34:52.599858 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 15629
Sep 24 11:34:52 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.3@KRBTEST.COM
I20260924 11:34:52.616901 15772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:52.600874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49542 (local address 127.6.212.3:37151)
0924 11:34:52.601510 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:52.601517 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:52.601547 (+ 30us) server_negotiation.cc:407] Connection header received
0924 11:34:52.601609 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:52.601615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:52.601824 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:52.602019 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:52.603347 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:52.604439 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:52.605713 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:52.606285 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:52.606448 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:52.611083 (+ 4635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:52.611094 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:52.611111 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:52.611153 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:52.614140 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:52.614796 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:52.614803 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:52.614805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:52.614940 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:52.615247 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:52.615250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:52.615252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:52.615683 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:52.615926 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:52.616307 (+ 381us) server_negotiation.cc:299] Negotiation successful
0924 11:34:52.616590 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":335,"threads_started":1}
I20260924 11:34:52.618355 15752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:34:52.702679 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 15629
I20260924 11:34:52.718712 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 15269
I20260924 11:34:52.720013 15331 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:34:52.823302 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 15269
I20260924 11:34:52.844990 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:35549
--webserver_interface=127.6.212.62
--webserver_port=45915
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:40115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:35549
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:34:53.178061 15775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:53.178428 15775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:53.178601 15775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:53.185947 15775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:34:53.186120 15775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:53.186228 15775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:53.186342 15775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:34:53.186429 15775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:34:53.196455 15775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:40115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:35549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:35549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.212.62
--webserver_port=45915
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:34:53.198263 15775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:53.199679 15775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:53.212251 15781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:53.212224 15780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:53.212165 15783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:53.212904 15775 server_base.cc:1034] running on GCE node
I20260924 11:34:53.213775 15775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:53.215293 15775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:53.216552 15775 hybrid_clock.cc:648] HybridClock initialized: now 1790249693216511 us; error 37 us; skew 500 ppm
I20260924 11:34:53.216917 15775 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:34:53 dist-test-slave-cq7c krb5kdc[15256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249693, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:53.220012 15775 init.cc:377] Logged in from keytab as kudu/127.6.212.62@KRBTEST.COM (short username kudu)
I20260924 11:34:53.221508 15775 webserver.cc:460] Webserver started at http://127.6.212.62:45915/ using document root <none> and password file <none>
I20260924 11:34:53.222234 15775 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:53.222401 15775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:53.228204 15775 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 11:34:53.231326 15790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:53.233018 15775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 11:34:53.233222 15775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "824c3ebb436840b79634de262ecfde40"
format_stamp: "Formatted at 2026-09-24 11:34:50 on dist-test-slave-cq7c"
server_key: "63fa24e1fa955ff787f2d5d5a5f80158"
server_key_iv: "7c1d8fe3a06c0565f1defc2de2abe0dc"
server_key_version: "encryptionkey@0"
I20260924 11:34:53.234228 15775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:53.249471 15775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:53.253810 15775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:34:53.254106 15775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:53.273679 15775 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.62:35549
I20260924 11:34:53.273702 15841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.62:35549 every 8 connection(s)
I20260924 11:34:53.275213 15775 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:34:53.278575 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 15775
I20260924 11:34:53.279340 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.1:45801
--local_ip_for_outbound_sockets=127.6.212.1
--tserver_master_addrs=127.6.212.62:35549
--webserver_port=43897
--webserver_interface=127.6.212.1
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab
--principal=kudu/127.6.212.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:40115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.212.1
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:34:53.282444 15842 sys_catalog.cc:263] Verifying existing consensus state
I20260924 11:34:53.285826 15842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 824c3ebb436840b79634de262ecfde40: Bootstrap starting.
I20260924 11:34:53.315079 15842 log.cc:824] T 00000000000000000000000000000000 P 824c3ebb436840b79634de262ecfde40: Log is configured to *not* fsync() on all Append() calls
I20260924 11:34:53.330981 15842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 824c3ebb436840b79634de262ecfde40: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 11:34:53.331694 15842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 824c3ebb436840b79634de262ecfde40: Bootstrap complete.
I20260924 11:34:53.345739 15842 raft_consensus.cc:348] T 00000000000000000000000000000000 P 824c3ebb436840b79634de262ecfde40 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824c3ebb436840b79634de262ecfde40" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 35549 } }
I20260924 11:34:53.346966 15842 raft_consensus.cc:729] T 00000000000000000000000000000000 P 824c3ebb436840b79634de262ecfde40 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 824c3ebb436840b79634de262ecfde40, State: Initialized, Role: FOLLOWER
I20260924 11:34:53.347744 15842 consensus_queue.cc:260] T 00000000000000000000000000000000 P 824c3ebb436840b79634de262ecfde40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824c3ebb436840b79634de262ecfde40" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 35549 } }
I20260924 11:34:53.348060 15842 raft_consensus.cc:388] T 00000000000000000000000000000000 P 824c3ebb436840b79634de262ecfde40 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:34:53.348203 15842 raft_consensus.cc:482] T 00000000000000000000000000000000 P 824c3ebb436840b79634de262ecfde40 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:34:53.348438 15842 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 824c3ebb436840b79634de262ecfde40 [term 1 FOLLOWER]: Advancing to term 2
I20260924 11:34:53.355130 15842 raft_consensus.cc:504] T 00000000000000000000000000000000 P 824c3ebb436840b79634de262ecfde40 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824c3ebb436840b79634de262ecfde40" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 35549 } }
I20260924 11:34:53.355795 15842 leader_election.cc:302] T 00000000000000000000000000000000 P 824c3ebb436840b79634de262ecfde40 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 824c3ebb436840b79634de262ecfde40; no voters:
I20260924 11:34:53.356441 15842 leader_election.cc:288] T 00000000000000000000000000000000 P 824c3ebb436840b79634de262ecfde40 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 11:34:53.356623 15846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 824c3ebb436840b79634de262ecfde40 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 11:34:53.357800 15846 raft_consensus.cc:686] T 00000000000000000000000000000000 P 824c3ebb436840b79634de262ecfde40 [term 2 LEADER]: Becoming Leader. State: Replica: 824c3ebb436840b79634de262ecfde40, State: Running, Role: LEADER
I20260924 11:34:53.358325 15846 consensus_queue.cc:237] T 00000000000000000000000000000000 P 824c3ebb436840b79634de262ecfde40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824c3ebb436840b79634de262ecfde40" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 35549 } }
I20260924 11:34:53.358587 15842 sys_catalog.cc:564] T 00000000000000000000000000000000 P 824c3ebb436840b79634de262ecfde40 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:34:53.360754 15848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 824c3ebb436840b79634de262ecfde40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 824c3ebb436840b79634de262ecfde40. Latest consensus state: current_term: 2 leader_uuid: "824c3ebb436840b79634de262ecfde40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824c3ebb436840b79634de262ecfde40" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 35549 } } }
I20260924 11:34:53.361123 15848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 824c3ebb436840b79634de262ecfde40 [sys.catalog]: This master's current role is: LEADER
I20260924 11:34:53.361613 15847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 824c3ebb436840b79634de262ecfde40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "824c3ebb436840b79634de262ecfde40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824c3ebb436840b79634de262ecfde40" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 35549 } } }
I20260924 11:34:53.361923 15847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 824c3ebb436840b79634de262ecfde40 [sys.catalog]: This master's current role is: LEADER
I20260924 11:34:53.369410 15852 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:34:53.375521 15852 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:34:53.376890 15852 catalog_manager.cc:1259] Loaded cluster ID: 443e8a6bb26c4a7ba34d29a2213decd3
I20260924 11:34:53.376988 15852 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:34:53.381089 15852 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:34:53.382237 15863 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 11:34:53.382804 15852 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 824c3ebb436840b79634de262ecfde40: Loaded TSK: 0
I20260924 11:34:53.384405 15852 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 11:34:53.646723 15844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:53.647064 15844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:53.647233 15844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:53.654850 15844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:53.655007 15844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:53.655176 15844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.1
I20260924 11:34:53.665498 15844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:40115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.1.keytab
--principal=oryx/127.6.212.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.1:45801
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.212.1
--webserver_port=43897
--tserver_master_addrs=127.6.212.62:35549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.1
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:34:53.666936 15844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:53.668305 15844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:53.683602 15869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:53.684163 15868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:53.684507 15844 server_base.cc:1034] running on GCE node
W20260924 11:34:53.684676 15871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:53.685396 15844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:53.686396 15844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:53.687626 15844 hybrid_clock.cc:648] HybridClock initialized: now 1790249693687563 us; error 56 us; skew 500 ppm
I20260924 11:34:53.687963 15844 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:34:53 dist-test-slave-cq7c krb5kdc[15256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249693, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.212.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:53.691128 15844 init.cc:377] Logged in from keytab as oryx/127.6.212.1@KRBTEST.COM (short username oryx)
I20260924 11:34:53.692703 15844 webserver.cc:460] Webserver started at http://127.6.212.1:43897/ using document root <none> and password file <none>
I20260924 11:34:53.693405 15844 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:53.693548 15844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:53.699579 15844 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 11:34:53.702826 15878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:53.704772 15844 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 11:34:53.704994 15844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-0/wal
uuid: "ff693c2e8abc426bb2d9b3fc69c71f3c"
format_stamp: "Formatted at 2026-09-24 11:34:51 on dist-test-slave-cq7c"
server_key: "b59b9d50f3318e7f4a164362775e0e1b"
server_key_iv: "32fc971e08724de783317c44ec983af4"
server_key_version: "encryptionkey@0"
I20260924 11:34:53.706784 15844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:53.730054 15844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:53.735172 15844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:34:53.735536 15844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:53.736615 15844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:34:53.738078 15844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:34:53.738224 15844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:53.738363 15844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:34:53.738457 15844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:53.773394 15844 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.1:45801
I20260924 11:34:53.773487 15990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.1:45801 every 8 connection(s)
I20260924 11:34:53.774955 15844 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
I20260924 11:34:53.778206 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 15844
I20260924 11:34:53.779034 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.2:41307
--local_ip_for_outbound_sockets=127.6.212.2
--tserver_master_addrs=127.6.212.62:35549
--webserver_port=35139
--webserver_interface=127.6.212.2
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab
--principal=kudu/127.6.212.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:40115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.212.2
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 11:34:53 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.1@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:34:53 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.1@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:34:53.795061 15994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:53.778239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.1:38187)
0924 11:34:53.779543 (+ 1304us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:53.779583 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:53.781412 (+ 1829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:53.782999 (+ 1587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:53.783012 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:53.783747 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:53.784302 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:53.784331 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:53.786041 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:53.786047 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:53.787440 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:53.787454 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:53.788574 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:53.788582 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:53.788893 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:53.789608 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:53.789631 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:53.793638 (+ 4007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":922,"thread_start_us":691,"threads_started":1}
W20260924 11:34:53.795092 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:34:53.778043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:38187 (local address 127.6.212.62:35549)
0924 11:34:53.778865 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:53.778882 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:53.779738 (+ 856us) server_negotiation.cc:407] Connection header received
0924 11:34:53.782067 (+ 2329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:53.782081 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:53.782443 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:53.782846 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:53.784524 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:53.785850 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:53.787587 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:53.788319 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:53.788540 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:53.794047 (+ 5507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.1:38187: BlockingRecv error: recv got EOF from 127.6.212.1:38187 (error 108)
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":366,"threads_started":1}
W20260924 11:34:53.795976 15991 heartbeater.cc:644] Failed to heartbeat to 127.6.212.62:35549 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.212.62:35549: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Sep 24 11:34:53 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.1@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:34:53 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.1@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:34:53.805862 15994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:53.796625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.1:35703)
0924 11:34:53.796872 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:53.796894 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:53.797018 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:53.797335 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:53.797341 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:53.797817 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:53.798066 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:53.798080 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:53.799101 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:53.799110 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:53.800866 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:53.800884 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:53.801612 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:53.801621 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:53.801842 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:53.802459 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:53.802481 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:53.805651 (+ 3170us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":59}
W20260924 11:34:53.806422 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:34:53.796887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:35703 (local address 127.6.212.62:35549)
0924 11:34:53.797035 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:53.797039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:53.797056 (+ 17us) server_negotiation.cc:407] Connection header received
0924 11:34:53.797132 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:53.797136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:53.797191 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:53.797268 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:53.798219 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:53.798954 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:53.801039 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:53.801455 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:53.801550 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:53.805919 (+ 4369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.1:35703: BlockingRecv error: recv got EOF from 127.6.212.1:35703 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 11:34:53 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.1@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:34:53 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.1@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:34:53.815413 15994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:53.806582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.1:50515)
0924 11:34:53.806761 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:53.806780 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:53.806885 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:53.807242 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:53.807247 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:53.807647 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:53.807824 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:53.807834 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:53.808914 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:53.808919 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:53.810366 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:53.810380 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:53.811053 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:53.811059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:53.811253 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:53.811847 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:53.811878 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:53.815073 (+ 3195us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":59}
W20260924 11:34:53.815582 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:34:53.806756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:50515 (local address 127.6.212.62:35549)
0924 11:34:53.806922 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:53.806928 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:53.806950 (+ 22us) server_negotiation.cc:407] Connection header received
0924 11:34:53.807004 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:53.807008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:53.807097 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:53.807200 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:53.807985 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:53.808752 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:53.810499 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:53.810911 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:53.811013 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:53.815280 (+ 4267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.1:50515: BlockingRecv error: recv got EOF from 127.6.212.1:50515 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260924 11:34:53.815693 15991 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 11:34:54.114464 15995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:54.114801 15995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:54.114967 15995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:54.122998 15995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:54.123139 15995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:54.123334 15995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.2
I20260924 11:34:54.133203 15995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:40115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.2.keytab
--principal=oryx/127.6.212.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.2:41307
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.212.2
--webserver_port=35139
--tserver_master_addrs=127.6.212.62:35549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.2
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:34:54.134548 15995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:54.135897 15995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:54.149756 16001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:54.149724 16003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:54.149794 16000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:54.153105 15995 server_base.cc:1034] running on GCE node
I20260924 11:34:54.153952 15995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:54.155066 15995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:54.156294 15995 hybrid_clock.cc:648] HybridClock initialized: now 1790249694156213 us; error 62 us; skew 500 ppm
I20260924 11:34:54.156605 15995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:34:54 dist-test-slave-cq7c krb5kdc[15256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249694, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.212.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:54.160058 15995 init.cc:377] Logged in from keytab as oryx/127.6.212.2@KRBTEST.COM (short username oryx)
I20260924 11:34:54.161574 15995 webserver.cc:460] Webserver started at http://127.6.212.2:35139/ using document root <none> and password file <none>
I20260924 11:34:54.162279 15995 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:54.162390 15995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:54.167847 15995 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 11:34:54.170655 16010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:54.172691 15995 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 11:34:54.172878 15995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-1/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-1/wal
uuid: "55d48eafb42543b78206c7a1b4c69f78"
format_stamp: "Formatted at 2026-09-24 11:34:51 on dist-test-slave-cq7c"
server_key: "403bcb3e0746a284724790bd98a97a12"
server_key_iv: "7f39f1d21cce3902e0ef5d5887807017"
server_key_version: "encryptionkey@0"
I20260924 11:34:54.174484 15995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:54.206329 15995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:54.210567 15995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:34:54.210904 15995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:54.212041 15995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:34:54.213425 15995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:34:54.213541 15995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:54.213673 15995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:34:54.213745 15995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:54.246671 15995 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.2:41307
I20260924 11:34:54.246762 16122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.2:41307 every 8 connection(s)
I20260924 11:34:54.248178 15995 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
I20260924 11:34:54.252863 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 15995
I20260924 11:34:54.253433 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.3:37151
--local_ip_for_outbound_sockets=127.6.212.3
--tserver_master_addrs=127.6.212.62:35549
--webserver_port=36137
--webserver_interface=127.6.212.3
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab
--principal=kudu/127.6.212.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:40115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.212.3
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 11:34:54 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.2@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:34:54 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.2@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 11:34:54.265151 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:34:54.250922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:58191 (local address 127.6.212.62:35549)
0924 11:34:54.251098 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:54.251102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:54.252392 (+ 1290us) server_negotiation.cc:407] Connection header received
0924 11:34:54.254492 (+ 2100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:54.254497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:54.254589 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:54.254701 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:54.256378 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:54.257084 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:54.258907 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:54.259331 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:54.259501 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:54.264837 (+ 5336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.2:58191: BlockingRecv error: recv got EOF from 127.6.212.2:58191 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 11:34:54.265729 16125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:54.251494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.2:58191)
0924 11:34:54.252224 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:54.252272 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:54.254207 (+ 1935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:54.254949 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:54.254958 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:54.255595 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:54.256197 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:54.256215 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:54.257247 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:54.257257 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:54.258761 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:54.258773 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:54.259693 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:54.259701 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:54.259912 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:54.260614 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:54.260636 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:54.264559 (+ 3923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":307,"threads_started":1}
W20260924 11:34:54.266497 16123 heartbeater.cc:644] Failed to heartbeat to 127.6.212.62:35549 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.212.62:35549: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Sep 24 11:34:54 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.2@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:34:54 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.2@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:34:54.277869 16125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:54.267131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.2:34857)
0924 11:34:54.267542 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:54.267563 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:54.267689 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:54.268076 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:54.268084 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:54.268550 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:54.268770 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:54.268784 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:54.269939 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:54.269944 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:54.271819 (+ 1875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:54.271837 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:54.272539 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:54.272548 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:54.272774 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:54.273567 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:54.273597 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:54.277585 (+ 3988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
W20260924 11:34:54.278188 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:34:54.267247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:34857 (local address 127.6.212.62:35549)
0924 11:34:54.267442 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:54.267450 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:54.267638 (+ 188us) server_negotiation.cc:407] Connection header received
0924 11:34:54.267857 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:54.267861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:54.267924 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:54.268006 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:54.268916 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:54.269786 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:54.271966 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:54.272371 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:54.272477 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:54.277877 (+ 5400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.2:34857: BlockingRecv error: recv got EOF from 127.6.212.2:34857 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 24 11:34:54 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.2@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:34:54 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.2@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:34:54.289132 16125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:54.278904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.2:41817)
0924 11:34:54.279137 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:54.279155 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:54.279260 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:54.279637 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:54.279642 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:54.280010 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:54.280224 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:54.280234 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:54.281988 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:54.281994 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:54.283571 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:54.283588 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:54.284627 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:54.284634 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:54.284817 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:54.285514 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:54.285535 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:54.288914 (+ 3379us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124}
W20260924 11:34:54.289382 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:34:54.279002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:41817 (local address 127.6.212.62:35549)
0924 11:34:54.279146 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:54.279152 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:54.279204 (+ 52us) server_negotiation.cc:407] Connection header received
0924 11:34:54.279401 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:54.279406 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:54.279487 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:54.279590 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:54.280387 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:54.281659 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:54.283724 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:54.284449 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:54.284606 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:54.289164 (+ 4558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.2:41817: BlockingRecv error: recv got EOF from 127.6.212.2:41817 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260924 11:34:54.289516 16123 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 11:34:54.594867 16126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:34:54.595329 16126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:34:54.595620 16126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:34:54.604080 16126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:34:54.604218 16126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:34:54.604332 16126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.3
I20260924 11:34:54.614846 16126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:40115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/krb5kdc/oryx_127.6.212.3.keytab
--principal=oryx/127.6.212.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.3:37151
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.212.3
--webserver_port=36137
--tserver_master_addrs=127.6.212.62:35549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.3
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:34:54.616250 16126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:34:54.617583 16126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:34:54.630890 16134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:54.630935 16132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:34:54.630851 16131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:34:54.631829 16126 server_base.cc:1034] running on GCE node
I20260924 11:34:54.632553 16126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:34:54.633565 16126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:34:54.634876 16126 hybrid_clock.cc:648] HybridClock initialized: now 1790249694634806 us; error 62 us; skew 500 ppm
I20260924 11:34:54.635226 16126 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:34:54 dist-test-slave-cq7c krb5kdc[15256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249694, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.212.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:34:54.638633 16126 init.cc:377] Logged in from keytab as oryx/127.6.212.3@KRBTEST.COM (short username oryx)
I20260924 11:34:54.640340 16126 webserver.cc:460] Webserver started at http://127.6.212.3:36137/ using document root <none> and password file <none>
I20260924 11:34:54.641072 16126 fs_manager.cc:359] Metadata directory not provided
I20260924 11:34:54.641232 16126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:34:54.647280 16126 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 11:34:54.650224 16141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:54.651888 16126 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:34:54.652083 16126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-2/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-2/wal
uuid: "5c83d66b3a8f4c2f825acd5dd8afc6dd"
format_stamp: "Formatted at 2026-09-24 11:34:52 on dist-test-slave-cq7c"
server_key: "d04de44921f2e6afb15b2b61cb3c777f"
server_key_iv: "0f5987a15fb1ff88dc55fe93ea56295b"
server_key_version: "encryptionkey@0"
I20260924 11:34:54.653084 16126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:34:54.678996 16126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:34:54.683799 16126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:34:54.684085 16126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:34:54.685066 16126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:34:54.686478 16126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:34:54.686606 16126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:54.686734 16126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:34:54.686829 16126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:34:54.721735 16126 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.3:37151
I20260924 11:34:54.721797 16253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.3:37151 every 8 connection(s)
I20260924 11:34:54.723253 16126 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
I20260924 11:34:54.729009 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 16126
Sep 24 11:34:54 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.3@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:34:54 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.3@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 11:34:54.740425 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:34:54.725876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:56005 (local address 127.6.212.62:35549)
0924 11:34:54.726123 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:54.726129 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:54.727548 (+ 1419us) server_negotiation.cc:407] Connection header received
0924 11:34:54.729892 (+ 2344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:54.729900 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:54.730003 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:54.730112 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:54.732008 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:54.732720 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:54.734295 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:54.734698 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:54.734836 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:54.740111 (+ 5275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.3:56005: BlockingRecv error: recv got EOF from 127.6.212.3:56005 (error 108)
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":79}
I20260924 11:34:54.741109 16256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:54.726512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.3:56005)
0924 11:34:54.727304 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:54.727342 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:54.729482 (+ 2140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:54.730469 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:54.730488 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:54.731178 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:54.731840 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:54.731859 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:54.732883 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:54.732888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:54.734153 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:54.734164 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:54.734939 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:54.734947 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:54.735238 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:54.736041 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:54.736082 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:54.739904 (+ 3822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":348,"threads_started":1}
W20260924 11:34:54.741811 16254 heartbeater.cc:644] Failed to heartbeat to 127.6.212.62:35549 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.212.62:35549: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
I20260924 11:34:54.745682 16258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:54.730167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59134 (local address 127.6.212.62:35549)
0924 11:34:54.730973 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:54.730979 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:54.730997 (+ 18us) server_negotiation.cc:407] Connection header received
0924 11:34:54.731134 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:54.731139 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:54.731201 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:54.731334 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:54.732802 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:54.733603 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:54.734866 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:54.735334 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:54.735523 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:54.738198 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:34:54.738207 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:34:54.738225 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:34:54.738309 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:34:54.741912 (+ 3603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:54.742778 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:54.742787 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:54.742797 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:54.743063 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:34:54.743515 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:34:54.743518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:34:54.743521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:34:54.744071 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:34:54.744454 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0924 11:34:54.744916 (+ 462us) server_negotiation.cc:299] Negotiation successful
0924 11:34:54.745343 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":694,"thread_start_us":516,"threads_started":1}
Sep 24 11:34:54 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.3@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:34:54 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.3@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:34:54.751271 16256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:54.742531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.3:51369)
0924 11:34:54.742741 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:54.742761 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:54.742852 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:54.743302 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:54.743306 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:54.743704 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:54.743839 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:54.743847 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:54.745402 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:54.745406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:54.746459 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:54.746471 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:54.747233 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:54.747239 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:54.747450 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:54.748018 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:54.748035 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:54.751064 (+ 3029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":63}
W20260924 11:34:54.751598 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:34:54.742660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:51369 (local address 127.6.212.62:35549)
0924 11:34:54.742968 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:54.742974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:54.742993 (+ 19us) server_negotiation.cc:407] Connection header received
0924 11:34:54.743052 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:54.743056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:54.743158 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:54.743260 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:54.743983 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:54.745246 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:54.746607 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:54.747088 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:54.747224 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:54.751309 (+ 4085us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.3:51369: BlockingRecv error: recv got EOF from 127.6.212.3:51369 (error 108)
Metrics: {"server-negotiator.queue_time_us":173}
Sep 24 11:34:54 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.3@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:34:54 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.3@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:34:54.760157 16256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:54.752053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.3:59201)
0924 11:34:54.752205 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:54.752221 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:54.752344 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:54.752674 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:54.752679 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:54.752989 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:54.753134 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:54.753141 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:54.754127 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:54.754135 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:54.755037 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:54.755048 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:54.755806 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:54.755811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:54.755990 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:54.756597 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:54.756612 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:54.759777 (+ 3165us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 11:34:54.760419 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:34:54.752139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:59201 (local address 127.6.212.62:35549)
0924 11:34:54.752424 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:54.752431 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:54.752450 (+ 19us) server_negotiation.cc:407] Connection header received
0924 11:34:54.752495 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:54.752499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:54.752549 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:54.752650 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:54.753268 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:54.753988 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:54.755183 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:54.755661 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:54.755794 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:54.760004 (+ 4210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.3:59201: BlockingRecv error: recv got EOF from 127.6.212.3:59201 (error 108)
Metrics: {"server-negotiator.queue_time_us":144}
W20260924 11:34:54.760731 16254 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 11:34:54 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.1@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:34:54 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.1@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:34:54.825568 16259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:54.816276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.1:42841)
0924 11:34:54.816741 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:54.816758 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:54.816890 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:54.817232 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:54.817237 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:54.817744 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:54.817967 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:54.817978 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:54.818968 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:54.818973 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:54.820108 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:54.820122 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:54.820936 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:54.820942 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:54.821091 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:54.821736 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:54.821755 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:54.825337 (+ 3582us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":251,"threads_started":1}
W20260924 11:34:54.825819 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:34:54.816354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:42841 (local address 127.6.212.62:35549)
0924 11:34:54.816582 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:54.816588 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:54.816848 (+ 260us) server_negotiation.cc:407] Connection header received
0924 11:34:54.816994 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:54.816998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:54.817058 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:54.817140 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:54.818103 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:54.818812 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:54.820242 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:54.820679 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:54.820770 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:54.825580 (+ 4810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.1:42841: BlockingRecv error: recv got EOF from 127.6.212.1:42841 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 24 11:34:55 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.2@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:34:55 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.2@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:34:55.299185 16260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:55.290189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.2:46341)
0924 11:34:55.290718 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:55.290738 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:55.290903 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:55.291261 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:55.291266 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:55.291809 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:55.291998 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:55.292006 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:55.293006 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:55.293012 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:55.293975 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:55.293985 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:55.294789 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:55.294794 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:55.294964 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:55.295651 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:55.295669 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:55.298851 (+ 3182us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":287,"threads_started":1}
W20260924 11:34:55.299292 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:34:55.290283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:46341 (local address 127.6.212.62:35549)
0924 11:34:55.290562 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:55.290570 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:55.290839 (+ 269us) server_negotiation.cc:407] Connection header received
0924 11:34:55.291028 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:55.291033 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:55.291109 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:55.291231 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:55.292142 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:55.292864 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:55.294133 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:55.294560 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:55.294744 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:55.299049 (+ 4305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.2:46341: BlockingRecv error: recv got EOF from 127.6.212.2:46341 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Sep 24 11:34:55 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.3@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:34:55 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.3@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:34:55.777284 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:55.761408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.3:48557)
0924 11:34:55.762000 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:55.762017 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:55.762135 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:55.762456 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:55.762460 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:55.762987 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:55.763185 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:55.763194 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:55.764595 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:55.764600 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:55.765606 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:55.765625 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:55.766508 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:55.766514 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:55.766653 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:55.773451 (+ 6798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:55.773473 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:55.776919 (+ 3446us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":323,"threads_started":1}
W20260924 11:34:55.777382 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:34:55.761459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:48557 (local address 127.6.212.62:35549)
0924 11:34:55.761662 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:55.761667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:55.762097 (+ 430us) server_negotiation.cc:407] Connection header received
0924 11:34:55.762242 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:55.762245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:55.762309 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:55.762402 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:55.763342 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:55.764421 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:55.765891 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:55.766317 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:55.766420 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:55.777135 (+ 10715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.3:48557: BlockingRecv error: recv got EOF from 127.6.212.3:48557 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 24 11:34:55 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.1@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:34:55 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.1@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:34:55.835696 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:55.826509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.1:45163)
0924 11:34:55.827142 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:55.827165 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:55.827336 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:55.827721 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:55.827724 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:55.828110 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:55.828256 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:55.828264 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:55.829406 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:55.829410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:55.830436 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:55.830448 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:55.831092 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:55.831097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:55.831212 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:55.831861 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:55.831879 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:55.835308 (+ 3429us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":355,"threads_started":1}
W20260924 11:34:55.835769 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:34:55.826572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:45163 (local address 127.6.212.62:35549)
0924 11:34:55.826803 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:55.826813 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:55.827259 (+ 446us) server_negotiation.cc:407] Connection header received
0924 11:34:55.827514 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:55.827517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:55.827589 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:55.827673 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:55.828441 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:55.829267 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:55.830563 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:55.830945 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:55.831046 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:55.835510 (+ 4464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.1:45163: BlockingRecv error: recv got EOF from 127.6.212.1:45163 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 24 11:34:56 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.2@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:34:56 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.2@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:34:56.309124 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:56.300453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.2:39489)
0924 11:34:56.300981 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:56.301001 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:56.301115 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:56.301461 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:56.301465 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:56.301849 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:56.301987 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:56.301995 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:56.302947 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:56.302951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:56.304096 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:56.304108 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:56.304819 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:56.304823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:56.304950 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:56.305620 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:56.305640 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:56.308877 (+ 3237us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":322,"threads_started":1}
W20260924 11:34:56.309365 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:34:56.300538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:39489 (local address 127.6.212.62:35549)
0924 11:34:56.300769 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:56.300779 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:56.301081 (+ 302us) server_negotiation.cc:407] Connection header received
0924 11:34:56.301214 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:56.301218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:56.301286 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:56.301373 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:56.302108 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:56.302805 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:56.304254 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:56.304662 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:56.304776 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:56.309113 (+ 4337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.2:39489: BlockingRecv error: recv got EOF from 127.6.212.2:39489 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 24 11:34:56 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.3@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:34:56 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.3@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:34:56.786739 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:56.778348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.3:47199)
0924 11:34:56.778926 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:56.778942 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:56.779057 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:56.779445 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:56.779449 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:56.779841 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:56.779981 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:56.779989 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:56.781104 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:56.781107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:56.782094 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:56.782106 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:56.782807 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:56.782811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:56.782929 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:56.783596 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:56.783614 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:56.786404 (+ 2790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":313,"threads_started":1}
W20260924 11:34:56.786943 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:34:56.778562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:47199 (local address 127.6.212.62:35549)
0924 11:34:56.778715 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:56.778720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:56.779015 (+ 295us) server_negotiation.cc:407] Connection header received
0924 11:34:56.779153 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:56.779157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:56.779216 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:56.779332 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:56.780118 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:56.780952 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:56.782237 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:56.782674 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:56.782774 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:56.786636 (+ 3862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.3:47199: BlockingRecv error: recv got EOF from 127.6.212.3:47199 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 24 11:34:56 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.1@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:34:56 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.1@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:34:56.845450 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:56.836867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.1:38257)
0924 11:34:56.837557 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:56.837578 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:56.837748 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:56.838121 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:56.838125 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:56.838448 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:56.838624 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:56.838632 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:56.839892 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:56.839896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:56.840793 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:56.840804 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:56.841435 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:56.841440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:56.841567 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:56.842215 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:56.842233 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:56.845197 (+ 2964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":471,"threads_started":1}
W20260924 11:34:56.845691 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:34:56.836941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:38257 (local address 127.6.212.62:35549)
0924 11:34:56.837131 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:56.837136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:56.837687 (+ 551us) server_negotiation.cc:407] Connection header received
0924 11:34:56.837891 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:56.837896 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:56.837978 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:56.838071 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:56.838872 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:56.839711 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:56.840922 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:56.841284 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:56.841383 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:56.845420 (+ 4037us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.1:38257: BlockingRecv error: recv got EOF from 127.6.212.1:38257 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 24 11:34:57 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.2@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:34:57 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.2@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:34:57.318670 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:57.310113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.2:40841)
0924 11:34:57.310637 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:57.310658 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:57.310768 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:57.311058 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:57.311062 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:57.311434 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:57.311591 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:57.311600 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:57.312600 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:57.312604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:57.313575 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:57.313586 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:57.314350 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:57.314355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:57.314474 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:57.315169 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:57.315185 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:57.318401 (+ 3216us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":310,"threads_started":1}
W20260924 11:34:57.318861 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:34:57.310333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:40841 (local address 127.6.212.62:35549)
0924 11:34:57.310512 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:57.310516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:57.310766 (+ 250us) server_negotiation.cc:407] Connection header received
0924 11:34:57.310865 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:57.310868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:57.310929 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:57.311011 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:57.311754 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:57.312479 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:57.313742 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:57.314200 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:57.314318 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:57.318592 (+ 4274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.2:40841: BlockingRecv error: recv got EOF from 127.6.212.2:40841 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 24 11:34:57 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.3@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:34:57 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.3@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:34:57.796913 16267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:57.787940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.3:41405)
0924 11:34:57.788449 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:57.788477 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:57.788600 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:57.788939 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:57.788942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:57.789271 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:57.789440 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:57.789447 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:57.790499 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:57.790502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:57.791527 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:57.791540 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:57.792306 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:57.792311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:57.792455 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:57.793116 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:57.793133 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:57.796624 (+ 3491us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":296,"threads_started":1}
W20260924 11:34:57.797020 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:34:57.788077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:41405 (local address 127.6.212.62:35549)
0924 11:34:57.788276 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:57.788280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:57.788559 (+ 279us) server_negotiation.cc:407] Connection header received
0924 11:34:57.788715 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:57.788719 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:57.788786 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:57.788882 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:57.789566 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:57.790352 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:57.791657 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:57.792139 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:57.792275 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:57.796811 (+ 4536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.3:41405: BlockingRecv error: recv got EOF from 127.6.212.3:41405 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 24 11:34:57 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.1@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:34:57 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.1@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:34:57.855269 16268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:57.846377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.1:58399)
0924 11:34:57.846883 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:57.846900 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:57.847012 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:57.847311 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:57.847314 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:57.847699 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:57.847846 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:57.847853 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:57.849034 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:57.849039 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:57.850149 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:57.850163 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:57.850863 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:57.850868 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:57.851010 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:57.851700 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:57.851719 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:57.854909 (+ 3190us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":352,"spinlock_wait_cycles":22784,"thread_start_us":260,"threads_started":1}
W20260924 11:34:57.855440 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:34:57.846466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:58399 (local address 127.6.212.62:35549)
0924 11:34:57.846728 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:57.846735 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:57.846982 (+ 247us) server_negotiation.cc:407] Connection header received
0924 11:34:57.847107 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:57.847111 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:57.847172 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:57.847272 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:57.848002 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:57.848872 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:57.850309 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:57.850719 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:57.850820 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:57.855135 (+ 4315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.1:58399: BlockingRecv error: recv got EOF from 127.6.212.1:58399 (error 108)
Metrics: {"server-negotiator.queue_time_us":126}
Sep 24 11:34:58 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.2@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:34:58 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.2@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:34:58.328437 16269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:58.319642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.2:50781)
0924 11:34:58.320169 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:58.320189 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:58.320379 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:58.320727 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:58.320793 (+ 66us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:58.321172 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:58.321320 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:58.321328 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:58.322342 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:58.322349 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:58.323334 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:58.323345 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:58.324057 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:58.324063 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:58.324255 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:58.325005 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:58.325030 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:58.328207 (+ 3177us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":299,"threads_started":1}
W20260924 11:34:58.328639 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:34:58.319739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:50781 (local address 127.6.212.62:35549)
0924 11:34:58.320056 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:58.320062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:58.320303 (+ 241us) server_negotiation.cc:407] Connection header received
0924 11:34:58.320524 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:58.320527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:58.320590 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:58.320692 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:58.321485 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:58.322200 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:58.323492 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:58.323907 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:58.324046 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:58.328401 (+ 4355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.2:50781: BlockingRecv error: recv got EOF from 127.6.212.2:50781 (error 108)
Metrics: {"server-negotiator.queue_time_us":160}
Sep 24 11:34:58 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.3@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:34:58 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.3@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:34:58.806759 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:58.797862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.3:33049)
0924 11:34:58.798387 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:58.798404 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:58.798520 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:58.798881 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:58.798885 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:58.799241 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:58.799462 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:58.799476 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:58.800550 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:58.800554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:58.801511 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:58.801522 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:58.802377 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:58.802381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:58.802508 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:58.803122 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:58.803139 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:58.806449 (+ 3310us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":305,"threads_started":1}
W20260924 11:34:58.806944 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:34:58.797981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:33049 (local address 127.6.212.62:35549)
0924 11:34:58.798158 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:58.798162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:58.798478 (+ 316us) server_negotiation.cc:407] Connection header received
0924 11:34:58.798647 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:58.798651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:58.798711 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:58.798801 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:58.799635 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:58.800430 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:58.801687 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:58.802228 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:58.802355 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:58.806659 (+ 4304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.3:33049: BlockingRecv error: recv got EOF from 127.6.212.3:33049 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 24 11:34:58 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.1@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:34:58 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.1@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:34:58.865002 16271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:58.856353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.1:34335)
0924 11:34:58.856930 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:58.856947 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:58.857068 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:58.857343 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:58.857346 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:58.857785 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:58.857993 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:58.858012 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:58.859117 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:58.859121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:58.860248 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:58.860260 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:58.860964 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:58.860968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:58.861090 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:58.861692 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:58.861708 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:58.864708 (+ 3000us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":248,"threads_started":1}
W20260924 11:34:58.865168 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:34:58.856512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:34335 (local address 127.6.212.62:35549)
0924 11:34:58.856726 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:58.856733 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:58.857037 (+ 304us) server_negotiation.cc:407] Connection header received
0924 11:34:58.857155 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:58.857158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:58.857218 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:58.857299 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:58.858129 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:58.858940 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:58.860385 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:58.860804 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:58.860909 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:58.864945 (+ 4036us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.1:34335: BlockingRecv error: recv got EOF from 127.6.212.1:34335 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 24 11:34:59 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.2@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:34:59 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.2@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:34:59.338187 16272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:59.329418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.2:39917)
0924 11:34:59.330066 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:59.330087 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:59.330204 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:59.330546 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:59.330550 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:59.330958 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:59.331117 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:59.331125 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:59.332163 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:59.332167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:59.333138 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:59.333150 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:59.333829 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:59.333834 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:59.333959 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:59.334542 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:59.334563 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:59.337843 (+ 3280us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":380,"threads_started":1}
W20260924 11:34:59.338425 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:34:59.329547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:39917 (local address 127.6.212.62:35549)
0924 11:34:59.329712 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:59.329717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:59.330193 (+ 476us) server_negotiation.cc:407] Connection header received
0924 11:34:59.330310 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:59.330314 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:59.330374 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:59.330457 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:59.331243 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:59.332043 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:59.333269 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:59.333675 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:59.333769 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:59.338110 (+ 4341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.2:39917: BlockingRecv error: recv got EOF from 127.6.212.2:39917 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 24 11:34:59 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.3@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:34:59 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.3@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:34:59.816989 16273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:59.807849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.3:58277)
0924 11:34:59.808453 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:59.808469 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:59.808635 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:59.809007 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:59.809012 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:59.809427 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:59.809585 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:59.809593 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:59.810870 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:59.810874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:59.812037 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:59.812050 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:59.812882 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:59.812887 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:59.813061 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:59.813737 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:59.813755 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:59.816733 (+ 2978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":277,"threads_started":1}
W20260924 11:34:59.817281 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:34:59.808073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:58277 (local address 127.6.212.62:35549)
0924 11:34:59.808355 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:59.808361 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:59.808564 (+ 203us) server_negotiation.cc:407] Connection header received
0924 11:34:59.808770 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:59.808776 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:59.808866 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:59.808981 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:59.809762 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:59.810658 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:59.812171 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:59.812705 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:59.812873 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:59.817014 (+ 4141us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.3:58277: BlockingRecv error: recv got EOF from 127.6.212.3:58277 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Sep 24 11:34:59 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.1@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:34:59 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.1@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:34:59.875691 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:34:59.866104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.1:53587)
0924 11:34:59.866666 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0924 11:34:59.866682 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:34:59.866837 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:34:59.867278 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:34:59.867283 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:34:59.867934 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:34:59.868109 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:59.868120 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:59.869281 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:59.869289 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:59.870379 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:34:59.870392 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:34:59.871173 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:34:59.871180 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:34:59.871428 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:59.872189 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:34:59.872218 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:34:59.875482 (+ 3264us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":317,"threads_started":1}
W20260924 11:34:59.875941 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:34:59.866221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:53587 (local address 127.6.212.62:35549)
0924 11:34:59.866507 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0924 11:34:59.866515 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:34:59.866776 (+ 261us) server_negotiation.cc:407] Connection header received
0924 11:34:59.867012 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:34:59.867017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:34:59.867099 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:34:59.867244 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:34:59.868305 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:59.869109 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:59.870542 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:34:59.870995 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:34:59.871161 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:34:59.875675 (+ 4514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.1:53587: BlockingRecv error: recv got EOF from 127.6.212.1:53587 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 24 11:35:00 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.2@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:35:00 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.2@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:35:00.348069 16275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:00.339311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.2:55145)
0924 11:35:00.339898 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:00.339920 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 11:35:00.340088 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:35:00.340703 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:35:00.340708 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:35:00.341122 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:35:00.341276 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:00.341284 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:00.342341 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:00.342345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:00.343253 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:00.343264 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:00.344007 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:00.344012 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:00.344142 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:00.344860 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:35:00.344879 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:35:00.347841 (+ 2962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":307,"threads_started":1}
W20260924 11:35:00.348407 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:35:00.339425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:55145 (local address 127.6.212.62:35549)
0924 11:35:00.339642 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:00.339652 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:00.340033 (+ 381us) server_negotiation.cc:407] Connection header received
0924 11:35:00.340401 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:00.340405 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:00.340502 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:00.340613 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:00.341462 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:00.342192 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:00.343428 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:00.343850 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:00.343960 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:00.348108 (+ 4148us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.2:55145: BlockingRecv error: recv got EOF from 127.6.212.2:55145 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 24 11:35:00 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.3@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:35:00 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.3@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:35:00.831764 16276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:00.818093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.3:53339)
0924 11:35:00.818664 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:00.818685 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 11:35:00.818845 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:35:00.819189 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:35:00.819194 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:35:00.819623 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:35:00.819773 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:00.819781 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:00.820843 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:00.820852 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:00.821795 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:00.821808 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:00.822580 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:00.822587 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:00.822772 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:00.828550 (+ 5778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:35:00.828572 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:35:00.831437 (+ 2865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":332,"threads_started":1}
W20260924 11:35:00.831944 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:35:00.818272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:53339 (local address 127.6.212.62:35549)
0924 11:35:00.818516 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:00.818523 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:00.818772 (+ 249us) server_negotiation.cc:407] Connection header received
0924 11:35:00.818974 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:00.818980 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:00.819051 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:00.819158 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:00.819945 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:00.820698 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:00.821953 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:00.822407 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:00.822543 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:00.831687 (+ 9144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.3:53339: BlockingRecv error: recv got EOF from 127.6.212.3:53339 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 24 11:35:00 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.1@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:35:00 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.1@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:35:00.885561 16277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:00.876662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.1:56281)
0924 11:35:00.877358 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:00.877377 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:35:00.877506 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:35:00.877818 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:35:00.877822 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:35:00.878278 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:35:00.878459 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:00.878468 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:00.879568 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:00.879573 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:00.880521 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:00.880534 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:00.881415 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:00.881420 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:00.881534 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:00.882196 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:35:00.882217 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:35:00.885287 (+ 3070us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":294,"threads_started":1}
W20260924 11:35:00.885758 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:35:00.876799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:56281 (local address 127.6.212.62:35549)
0924 11:35:00.877058 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:00.877066 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:00.877467 (+ 401us) server_negotiation.cc:407] Connection header received
0924 11:35:00.877612 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:00.877616 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:00.877678 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:00.877758 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:00.878606 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:00.879409 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:00.880800 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:00.881245 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:00.881349 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:00.885520 (+ 4171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.1:56281: BlockingRecv error: recv got EOF from 127.6.212.1:56281 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 24 11:35:01 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.2@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:35:01 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.2@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:35:01.358937 16278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:01.349247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.2:34207)
0924 11:35:01.349960 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:01.349981 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 11:35:01.350108 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:35:01.350450 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:35:01.350453 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:35:01.350907 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:35:01.351081 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:01.351090 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:01.352227 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:01.352233 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:01.353660 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:01.353676 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:01.354419 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:01.354424 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:01.354533 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:01.355184 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:35:01.355201 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:35:01.358649 (+ 3448us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":305,"threads_started":1}
W20260924 11:35:01.359236 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:35:01.349378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:34207 (local address 127.6.212.62:35549)
0924 11:35:01.349657 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:01.349662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:01.350071 (+ 409us) server_negotiation.cc:407] Connection header received
0924 11:35:01.350210 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:01.350214 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:01.350305 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:01.350403 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:01.351226 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:01.352076 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:01.353803 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:01.354277 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:01.354388 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:01.358909 (+ 4521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.2:34207: BlockingRecv error: recv got EOF from 127.6.212.2:34207 (error 108)
Metrics: {"server-negotiator.queue_time_us":141}
Sep 24 11:35:01 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.3@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:35:01 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.3@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:35:01.841794 16279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:01.832905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.3:32799)
0924 11:35:01.833518 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:01.833533 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:35:01.833649 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:35:01.834028 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:35:01.834031 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:35:01.834431 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:35:01.834589 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:01.834597 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:01.835660 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:01.835664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:01.836561 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:01.836571 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:01.837276 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:01.837280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:01.837390 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:01.838070 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:35:01.838086 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:35:01.841401 (+ 3315us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":286,"threads_started":1}
W20260924 11:35:01.841877 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:35:01.832987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:32799 (local address 127.6.212.62:35549)
0924 11:35:01.833163 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:01.833167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:01.833613 (+ 446us) server_negotiation.cc:407] Connection header received
0924 11:35:01.833815 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:01.833819 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:01.833881 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:01.833969 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:01.834748 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:01.835522 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:01.836691 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:01.837131 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:01.837240 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:01.841646 (+ 4406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.3:32799: BlockingRecv error: recv got EOF from 127.6.212.3:32799 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 24 11:35:01 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.1@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:35:01 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.1@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:35:01.896155 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:01.886828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.1:54157)
0924 11:35:01.887430 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:01.887447 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:35:01.887586 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:35:01.887961 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:35:01.887964 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:35:01.888353 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:35:01.888508 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:01.888516 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:01.889587 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:01.889591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:01.890687 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:01.890701 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:01.891807 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:01.891813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:01.892022 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:01.892679 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:35:01.892701 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:35:01.895923 (+ 3222us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":317,"threads_started":1}
W20260924 11:35:01.896377 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:35:01.887030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:54157 (local address 127.6.212.62:35549)
0924 11:35:01.887205 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:01.887209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:01.887585 (+ 376us) server_negotiation.cc:407] Connection header received
0924 11:35:01.887743 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:01.887747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:01.887813 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:01.887909 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:01.888703 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:01.889440 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:01.890888 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:01.891592 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:01.891773 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:01.896136 (+ 4363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.1:54157: BlockingRecv error: recv got EOF from 127.6.212.1:54157 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 24 11:35:02 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.2@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:35:02 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.2@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:35:02.368709 16281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:02.359996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.2:45083)
0924 11:35:02.360569 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:02.360591 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 11:35:02.360714 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:35:02.361007 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:35:02.361011 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:35:02.361398 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:35:02.361548 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:02.361555 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:02.362757 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:02.362761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:02.363764 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:02.363776 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:02.364566 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:02.364571 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:02.364691 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:02.365368 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:35:02.365385 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:35:02.368320 (+ 2935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":278,"threads_started":1}
W20260924 11:35:02.368829 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:35:02.360140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:45083 (local address 127.6.212.62:35549)
0924 11:35:02.360361 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:02.360368 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:02.360677 (+ 309us) server_negotiation.cc:407] Connection header received
0924 11:35:02.360812 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:02.360816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:02.360879 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:02.360963 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:02.361673 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:02.362608 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:02.363953 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:02.364396 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:02.364503 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:02.368547 (+ 4044us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.2:45083: BlockingRecv error: recv got EOF from 127.6.212.2:45083 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 24 11:35:02 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.3@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:35:02 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.3@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:35:02.851486 16282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:02.842774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.3:46469)
0924 11:35:02.843347 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:02.843417 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0924 11:35:02.843536 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:35:02.843828 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:35:02.843831 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:35:02.844223 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:35:02.844405 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:02.844413 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:02.845410 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:02.845413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:02.846323 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:02.846334 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:02.847057 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:02.847061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:02.847179 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:02.847959 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:35:02.847977 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:35:02.851144 (+ 3167us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":304,"threads_started":1}
W20260924 11:35:02.851827 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:35:02.842923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:46469 (local address 127.6.212.62:35549)
0924 11:35:02.843111 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:02.843115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:02.843498 (+ 383us) server_negotiation.cc:407] Connection header received
0924 11:35:02.843634 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:02.843637 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:02.843697 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:02.843782 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:02.844546 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:02.845256 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:02.846444 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:02.846889 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:02.846983 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:02.851473 (+ 4490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.3:46469: BlockingRecv error: recv got EOF from 127.6.212.3:46469 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 24 11:35:02 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.1@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:35:02 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.1@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:35:02.905972 16283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:02.897232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.1:43027)
0924 11:35:02.897788 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:02.897806 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:35:02.897974 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:35:02.898344 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:35:02.898349 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:35:02.898829 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:35:02.898998 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:02.899006 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:02.900074 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:02.900080 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:02.901160 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:02.901172 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:02.902014 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:02.902019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:02.902191 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:02.902798 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:35:02.902816 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:35:02.905605 (+ 2789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":294,"threads_started":1}
W20260924 11:35:02.906038 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:35:02.897330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:43027 (local address 127.6.212.62:35549)
0924 11:35:02.897585 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:02.897593 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:02.897893 (+ 300us) server_negotiation.cc:407] Connection header received
0924 11:35:02.898098 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:02.898104 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:02.898203 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:02.898312 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:02.899140 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:02.899937 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:02.901300 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:02.901860 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:02.902003 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:02.905807 (+ 3804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.1:43027: BlockingRecv error: recv got EOF from 127.6.212.1:43027 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 24 11:35:03 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.2@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:35:03 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.2@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:35:03.379021 16284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:03.369757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.2:55871)
0924 11:35:03.370247 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:03.370267 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 11:35:03.370385 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:35:03.370843 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:35:03.370846 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:35:03.371212 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:35:03.371456 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:03.371468 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:03.372547 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:03.372552 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:03.373675 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:03.373690 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:03.374393 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:03.374398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:03.374528 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:03.375096 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:35:03.375112 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:35:03.378661 (+ 3549us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":274,"threads_started":1}
W20260924 11:35:03.379168 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:35:03.369776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:55871 (local address 127.6.212.62:35549)
0924 11:35:03.369997 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:03.370007 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:03.370357 (+ 350us) server_negotiation.cc:407] Connection header received
0924 11:35:03.370555 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:03.370559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:03.370620 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:03.370675 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:03.371600 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:03.372393 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:03.373813 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:03.374242 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:03.374342 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:03.378891 (+ 4549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.2:55871: BlockingRecv error: recv got EOF from 127.6.212.2:55871 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 24 11:35:03 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.3@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:35:03 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.3@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:35:03.861533 16285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:03.852568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.3:50399)
0924 11:35:03.853106 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:03.853121 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:35:03.853236 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:35:03.853660 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:35:03.853663 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:35:03.854029 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:35:03.854184 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:03.854191 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:03.855246 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:03.855250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:03.856295 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:03.856307 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:03.857287 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:03.857292 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:03.857418 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:03.858156 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:35:03.858184 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:35:03.861273 (+ 3089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":288,"threads_started":1}
W20260924 11:35:03.861819 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:35:03.852626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:50399 (local address 127.6.212.62:35549)
0924 11:35:03.852826 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:03.852831 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:03.853216 (+ 385us) server_negotiation.cc:407] Connection header received
0924 11:35:03.853461 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:03.853465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:03.853534 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:03.853620 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:03.854315 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:03.855122 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:03.856429 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:03.857088 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:03.857225 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:03.861480 (+ 4255us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.3:50399: BlockingRecv error: recv got EOF from 127.6.212.3:50399 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 24 11:35:03 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.1@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:35:03 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.1@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:35:03.915197 16286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:03.907042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.1:46635)
0924 11:35:03.907620 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:03.907642 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 11:35:03.907797 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:35:03.908122 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:35:03.908127 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:35:03.908509 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:35:03.908700 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:03.908709 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:03.909746 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:03.909753 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:03.910659 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:03.910669 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:03.911402 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:03.911407 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:03.911578 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:03.912131 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:35:03.912149 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:35:03.914875 (+ 2726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":348,"threads_started":1}
W20260924 11:35:03.915328 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:35:03.907177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:46635 (local address 127.6.212.62:35549)
0924 11:35:03.907437 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:03.907445 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:03.907733 (+ 288us) server_negotiation.cc:407] Connection header received
0924 11:35:03.907917 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:03.907922 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:03.907992 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:03.908092 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:03.908852 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:03.909608 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:03.910793 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:03.911233 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:03.911368 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:03.915048 (+ 3680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.1:46635: BlockingRecv error: recv got EOF from 127.6.212.1:46635 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 24 11:35:04 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.2@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:35:04 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.2@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:35:04.388156 16287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:04.379889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.2:49187)
0924 11:35:04.380407 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:04.380429 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 11:35:04.380537 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:35:04.380980 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:35:04.380983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:35:04.381343 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:35:04.381565 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:04.381575 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:04.382782 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:04.382786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:04.383756 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:04.383767 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:04.384387 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:04.384391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:04.384506 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:04.385067 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:35:04.385083 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:35:04.387922 (+ 2839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":307,"threads_started":1}
W20260924 11:35:04.388415 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:35:04.379987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:49187 (local address 127.6.212.62:35549)
0924 11:35:04.380164 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:04.380169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:04.380500 (+ 331us) server_negotiation.cc:407] Connection header received
0924 11:35:04.380688 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:04.380693 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:04.380826 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:04.380946 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:04.381919 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:04.382659 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:04.383869 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:04.384264 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:04.384356 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:04.388146 (+ 3790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.2:49187: BlockingRecv error: recv got EOF from 127.6.212.2:49187 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 11:35:04 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.3@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:35:04 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.3@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:35:04.870878 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:04.862391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.3:53283)
0924 11:35:04.862904 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:04.862924 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 11:35:04.863087 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:35:04.863443 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:35:04.863447 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:35:04.863864 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:35:04.864000 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:04.864008 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:04.865017 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:04.865025 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:04.865941 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:04.865952 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:04.866660 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:04.866666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:04.866846 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:04.867552 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:35:04.867569 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:35:04.870655 (+ 3086us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":281,"threads_started":1}
W20260924 11:35:04.871161 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:35:04.862593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:53283 (local address 127.6.212.62:35549)
0924 11:35:04.862906 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:04.862913 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:04.863015 (+ 102us) server_negotiation.cc:407] Connection header received
0924 11:35:04.863213 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:04.863224 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:04.863287 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:04.863424 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:04.864149 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:04.864885 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:04.866081 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:04.866498 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:04.866626 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:04.870869 (+ 4243us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.3:53283: BlockingRecv error: recv got EOF from 127.6.212.3:53283 (error 108)
Metrics: {"server-negotiator.queue_time_us":138}
Sep 24 11:35:04 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.1@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:35:04 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.1@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:35:04.924479 16289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:04.916271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.1:53565)
0924 11:35:04.916766 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:04.916782 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:35:04.916946 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:35:04.917289 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:35:04.917295 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:35:04.917754 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:35:04.917928 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:04.917936 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:04.918941 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:04.918949 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:04.919895 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:04.919906 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:04.920604 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:04.920609 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:04.920788 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:04.921333 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:35:04.921364 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:35:04.924232 (+ 2868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":265,"threads_started":1}
W20260924 11:35:04.924758 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:35:04.916294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:53565 (local address 127.6.212.62:35549)
0924 11:35:04.916496 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:04.916503 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:04.916868 (+ 365us) server_negotiation.cc:407] Connection header received
0924 11:35:04.917063 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:04.917069 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:04.917153 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:04.917257 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:04.918066 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:04.918812 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:04.920027 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:04.920457 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:04.920591 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:04.924404 (+ 3813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.1:53565: BlockingRecv error: recv got EOF from 127.6.212.1:53565 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 24 11:35:05 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.2@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:35:05 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.2@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:35:05.398636 16290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:05.389127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.2:44667)
0924 11:35:05.389727 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:05.389752 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 11:35:05.389900 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:35:05.390339 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:35:05.390344 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:35:05.390953 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:35:05.391207 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:05.391216 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:05.392438 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:05.392447 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:05.393435 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:05.393446 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:05.394238 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:05.394245 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:05.394439 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:05.395141 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:35:05.395158 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:35:05.398235 (+ 3077us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":255,"threads_started":1}
W20260924 11:35:05.398751 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:35:05.389200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:44667 (local address 127.6.212.62:35549)
0924 11:35:05.389389 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:05.389396 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:05.389831 (+ 435us) server_negotiation.cc:407] Connection header received
0924 11:35:05.390077 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:05.390085 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:05.390182 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:05.390310 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:05.391354 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:05.392293 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:05.393587 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:05.394055 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:05.394223 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:05.398481 (+ 4258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.2:44667: BlockingRecv error: recv got EOF from 127.6.212.2:44667 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 24 11:35:05 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.3@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:35:05 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.3@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:35:05.886497 16291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:05.871839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.3:36631)
0924 11:35:05.872433 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:05.872449 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:35:05.872572 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:35:05.872901 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:35:05.872904 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:35:05.873309 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:35:05.873513 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:05.873522 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:05.874528 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:05.874535 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:05.875759 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:05.875774 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:05.876531 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:05.876537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:05.876799 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:05.882895 (+ 6096us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:35:05.882915 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:35:05.886216 (+ 3301us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":340,"threads_started":1}
W20260924 11:35:05.886674 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:35:05.871964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:36631 (local address 127.6.212.62:35549)
0924 11:35:05.872130 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:05.872135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:05.872525 (+ 390us) server_negotiation.cc:407] Connection header received
0924 11:35:05.872699 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:05.872702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:05.872761 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:05.872854 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:05.873640 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:05.874363 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:05.875927 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:05.876368 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:05.876466 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:05.886439 (+ 9973us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.3:36631: BlockingRecv error: recv got EOF from 127.6.212.3:36631 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 24 11:35:05 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.1@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:35:05 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.1@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:35:05.934005 16292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:05.925465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.1:38315)
0924 11:35:05.926052 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:05.926068 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:35:05.926202 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:35:05.926525 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:35:05.926528 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:35:05.926908 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:35:05.927057 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:05.927064 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:05.928153 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:05.928157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:05.929073 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:05.929083 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:05.929813 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:05.929817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:05.929986 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:05.930682 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:35:05.930699 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:35:05.933774 (+ 3075us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":343,"threads_started":1}
W20260924 11:35:05.934312 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:35:05.925606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:38315 (local address 127.6.212.62:35549)
0924 11:35:05.925847 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:05.925852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:05.926143 (+ 291us) server_negotiation.cc:407] Connection header received
0924 11:35:05.926310 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:05.926314 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:05.926379 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:05.926466 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:05.927226 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:05.928007 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:05.929189 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:05.929652 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:05.929762 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:05.933995 (+ 4233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.1:38315: BlockingRecv error: recv got EOF from 127.6.212.1:38315 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
Sep 24 11:35:06 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.2@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:35:06 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.2@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:35:06.408659 16293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:06.399728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.2:59367)
0924 11:35:06.400289 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:06.400314 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 11:35:06.400445 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:35:06.400781 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:35:06.400785 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:35:06.401153 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:35:06.401302 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:06.401310 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:06.402487 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:06.402491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:06.403575 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:06.403587 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:06.404314 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:06.404319 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:06.404448 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:06.405022 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:35:06.405038 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:35:06.408437 (+ 3399us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":309,"threads_started":1}
W20260924 11:35:06.408995 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:35:06.399775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:59367 (local address 127.6.212.62:35549)
0924 11:35:06.400030 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:06.400039 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:06.400405 (+ 366us) server_negotiation.cc:407] Connection header received
0924 11:35:06.400551 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:06.400554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:06.400621 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:06.400699 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:06.401477 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:06.402343 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:06.403747 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:06.404153 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:06.404262 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:06.408651 (+ 4389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.2:59367: BlockingRecv error: recv got EOF from 127.6.212.2:59367 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 24 11:35:06 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.3@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:35:06 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.3@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:35:06.896486 16294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:06.887724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.3:37811)
0924 11:35:06.888262 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:06.888280 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:35:06.888415 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:35:06.888816 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:35:06.888819 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:35:06.889202 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:35:06.889350 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:06.889358 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:06.890619 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:06.890625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:06.891852 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:06.891864 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:06.892557 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:06.892562 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:06.892755 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:06.893350 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:35:06.893366 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:35:06.896252 (+ 2886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":305,"threads_started":1}
W20260924 11:35:06.896823 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:35:06.887797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:37811 (local address 127.6.212.62:35549)
0924 11:35:06.888057 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:06.888065 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:06.888380 (+ 315us) server_negotiation.cc:407] Connection header received
0924 11:35:06.888525 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:06.888529 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:06.888592 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:06.888760 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:06.889512 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:06.890437 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:06.891980 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:06.892428 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:06.892526 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:06.896471 (+ 3945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.3:37811: BlockingRecv error: recv got EOF from 127.6.212.3:37811 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 24 11:35:06 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.1@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:35:06 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.1@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:35:06.943872 16295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:06.934967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.1:37745)
0924 11:35:06.935551 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:06.935568 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:35:06.935805 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:35:06.936143 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:35:06.936148 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:35:06.936546 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:35:06.936706 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:06.936715 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:06.937792 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:06.937799 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:06.939065 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:06.939076 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:06.940085 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:06.940092 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:06.940295 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:06.940906 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:35:06.940922 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:35:06.943644 (+ 2722us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":299,"threads_started":1}
W20260924 11:35:06.944072 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:35:06.935085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:37745 (local address 127.6.212.62:35549)
0924 11:35:06.935330 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:06.935338 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:06.935726 (+ 388us) server_negotiation.cc:407] Connection header received
0924 11:35:06.935930 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:06.935939 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:06.936003 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:06.936143 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:06.936849 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:06.937657 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:06.939196 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:06.939844 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:06.939972 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:06.943823 (+ 3851us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.1:37745: BlockingRecv error: recv got EOF from 127.6.212.1:37745 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 24 11:35:07 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.2@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:35:07 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.2@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:35:07.419163 16296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:07.409793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.2:42573)
0924 11:35:07.410393 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:07.410413 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 11:35:07.410528 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:35:07.410914 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:35:07.410917 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:35:07.411304 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:35:07.411531 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:07.411540 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:07.412691 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:07.412695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:07.413690 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:07.413723 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:07.414492 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:07.414496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:07.414607 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:07.415278 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:35:07.415294 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:35:07.418828 (+ 3534us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":294,"threads_started":1}
W20260924 11:35:07.419276 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:35:07.409935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:42573 (local address 127.6.212.62:35549)
0924 11:35:07.410149 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:07.410154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:07.410492 (+ 338us) server_negotiation.cc:407] Connection header received
0924 11:35:07.410698 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:07.410703 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:07.410771 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:07.410868 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:07.411703 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:07.412526 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:07.413858 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:07.414337 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:07.414448 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:07.419026 (+ 4578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.2:42573: BlockingRecv error: recv got EOF from 127.6.212.2:42573 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 24 11:35:07 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.3@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:35:07 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.3@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:35:07.907125 16297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:07.897533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.3:60763)
0924 11:35:07.898100 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:07.898116 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:35:07.898343 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:35:07.898720 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:35:07.898727 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:35:07.899167 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:35:07.899331 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:07.899340 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:07.900461 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:07.900467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:07.901475 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:07.901487 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:07.902420 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:07.902426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:07.902594 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:07.903279 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:35:07.903296 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:35:07.906758 (+ 3462us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":301,"threads_started":1}
W20260924 11:35:07.907279 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:35:07.897616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:60763 (local address 127.6.212.62:35549)
0924 11:35:07.897894 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:07.897902 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:07.898253 (+ 351us) server_negotiation.cc:407] Connection header received
0924 11:35:07.898482 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:07.898490 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:07.898570 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:07.898683 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:07.899545 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:07.900317 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:07.901658 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:07.902237 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:07.902375 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:07.907003 (+ 4628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.3:60763: BlockingRecv error: recv got EOF from 127.6.212.3:60763 (error 108)
Metrics: {"server-negotiator.queue_time_us":123}
Sep 24 11:35:07 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.1@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:35:07 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.1@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:35:07.953864 16298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:07.944920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.1:51235)
0924 11:35:07.945447 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:07.945464 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:35:07.945623 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:35:07.945992 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:35:07.945996 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:35:07.946388 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:35:07.946561 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:07.946569 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:07.947715 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:07.947721 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:07.948633 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:07.948643 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:07.949359 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:07.949364 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:07.949539 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:07.950220 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:35:07.950238 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:35:07.953504 (+ 3266us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":281,"threads_started":1}
W20260924 11:35:07.953996 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:35:07.944994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:51235 (local address 127.6.212.62:35549)
0924 11:35:07.945365 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:07.945372 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:07.945545 (+ 173us) server_negotiation.cc:407] Connection header received
0924 11:35:07.945752 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:07.945758 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:07.945834 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:07.945941 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:07.946710 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:07.947575 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:07.948775 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:07.949199 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:07.949346 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:07.953721 (+ 4375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.1:51235: BlockingRecv error: recv got EOF from 127.6.212.1:51235 (error 108)
Metrics: {"server-negotiator.queue_time_us":204}
Sep 24 11:35:08 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.2@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:35:08 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.2@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:35:08.429006 16299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:08.420314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.2:58827)
0924 11:35:08.420816 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:08.420840 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 11:35:08.420965 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:35:08.421351 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:35:08.421354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:35:08.421726 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:35:08.421901 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:08.421910 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:08.422989 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:08.422993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:08.423979 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:08.423990 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:08.424682 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:08.424686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:08.424819 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:08.425491 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:35:08.425509 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:35:08.428758 (+ 3249us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":238,"threads_started":1}
W20260924 11:35:08.429188 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:35:08.420556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:58827 (local address 127.6.212.62:35549)
0924 11:35:08.420799 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:08.420811 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:08.420945 (+ 134us) server_negotiation.cc:407] Connection header received
0924 11:35:08.421099 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:08.421103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:08.421181 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:08.421279 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:08.422038 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:08.422848 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:08.424106 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:08.424546 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:08.424646 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:08.428956 (+ 4310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.2:58827: BlockingRecv error: recv got EOF from 127.6.212.2:58827 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 24 11:35:08 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.3@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:35:08 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.3@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:35:08.917393 16300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:08.908215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.3:35755)
0924 11:35:08.908766 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:08.908783 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:35:08.908948 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:35:08.909365 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:35:08.909369 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:35:08.909899 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:35:08.910099 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:08.910112 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:08.911421 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:08.911429 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:08.912402 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:08.912419 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:08.913121 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:08.913126 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:08.913254 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:08.914055 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:35:08.914073 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:35:08.917020 (+ 2947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":281,"threads_started":1}
W20260924 11:35:08.917496 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:35:08.908297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:35755 (local address 127.6.212.62:35549)
0924 11:35:08.908475 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:08.908479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:08.908882 (+ 403us) server_negotiation.cc:407] Connection header received
0924 11:35:08.909076 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:08.909083 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:08.909173 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:08.909299 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:08.910289 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:08.911196 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:08.912568 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:08.912959 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:08.913059 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:08.917261 (+ 4202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.3:35755: BlockingRecv error: recv got EOF from 127.6.212.3:35755 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 24 11:35:08 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.1@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:35:08 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.1@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:35:08.964847 16301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:08.955210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.1:50857)
0924 11:35:08.955742 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:08.955757 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:35:08.955917 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:35:08.956271 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:35:08.956275 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:35:08.956758 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:35:08.956941 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:08.956950 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:08.958128 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:08.958134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:08.959331 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:08.959347 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:08.960417 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:08.960421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:08.960536 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:08.961164 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:35:08.961217 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:35:08.964523 (+ 3306us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":308,"threads_started":1}
W20260924 11:35:08.964918 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:35:08.955295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:50857 (local address 127.6.212.62:35549)
0924 11:35:08.955576 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:08.955583 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:08.955845 (+ 262us) server_negotiation.cc:407] Connection header received
0924 11:35:08.956055 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:08.956059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:08.956118 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:08.956265 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:08.957097 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:08.957990 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:08.959513 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:08.960064 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:08.960186 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:08.964707 (+ 4521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.1:50857: BlockingRecv error: recv got EOF from 127.6.212.1:50857 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
Sep 24 11:35:09 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.2@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 11:35:09 dist-test-slave-cq7c krb5kdc[15256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.212.2@KRBTEST.COM for oryx/127.6.212.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 11:35:09.453169 16302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:09.430408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35549 (local address 127.6.212.2:42351)
0924 11:35:09.431033 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:09.431053 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 11:35:09.431215 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:35:09.431722 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:35:09.431727 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:35:09.432294 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:35:09.432572 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:09.432588 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:09.433898 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:09.433905 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:09.434987 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:09.434999 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:09.435960 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:09.435965 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:09.436159 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:09.449761 (+ 13602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:35:09.449790 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:35:09.452902 (+ 3112us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.212.62:35549: Server oryx/127.6.212.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":423,"threads_started":1}
W20260924 11:35:09.453466 15993 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:35:09.430549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:42351 (local address 127.6.212.62:35549)
0924 11:35:09.430786 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:09.430796 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:09.431151 (+ 355us) server_negotiation.cc:407] Connection header received
0924 11:35:09.431405 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:09.431411 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:09.431499 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:09.431605 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:09.432719 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:09.433745 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:09.435154 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:09.435798 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:09.435951 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:09.453168 (+ 17217us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.212.2:42351: BlockingRecv error: recv got EOF from 127.6.212.2:42351 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
I20260924 11:35:09.731213 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 15844
I20260924 11:35:09.744935 16304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:09.732604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43560 (local address 127.6.212.1:45801)
0924 11:35:09.733328 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:09.733336 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:09.733365 (+ 29us) server_negotiation.cc:407] Connection header received
0924 11:35:09.733428 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:09.733435 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:09.733755 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:09.733944 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:09.735699 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:09.736784 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:09.738268 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:09.738850 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:09.738956 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:09.741815 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:35:09.741823 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:35:09.741839 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:35:09.741873 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:35:09.744406 (+ 2533us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.212.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 11:35:09.744678 (+ 272us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43560: Request ticket server kudu/127.6.212.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":426,"threads_started":1}
W20260924 11:35:09.745077 16304 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43560: Request ticket server kudu/127.6.212.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 11:35:09.745262 6992 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 15844: Not authorized: Client connection negotiation failed: client connection to 127.6.212.1:45801: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.212.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 11:35:09.745446 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 15844
I20260924 11:35:09.762696 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 15995
I20260924 11:35:09.775693 16305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:09.763890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48372 (local address 127.6.212.2:41307)
0924 11:35:09.764560 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:09.764568 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:09.764601 (+ 33us) server_negotiation.cc:407] Connection header received
0924 11:35:09.764672 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:09.764679 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:09.764901 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:09.765071 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:09.766477 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:09.767731 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:09.769045 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:09.769703 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:09.769855 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:09.772492 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:35:09.772502 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:35:09.772517 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:35:09.772557 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:35:09.774988 (+ 2431us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.212.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 11:35:09.775277 (+ 289us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48372: Request ticket server kudu/127.6.212.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":347,"threads_started":1}
W20260924 11:35:09.775884 16305 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48372: Request ticket server kudu/127.6.212.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 11:35:09.775986 6992 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 15995: Not authorized: Client connection negotiation failed: client connection to 127.6.212.2:41307: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.212.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 11:35:09.776178 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 15995
I20260924 11:35:09.793183 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 16126
I20260924 11:35:09.806391 16306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:09.794199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36564 (local address 127.6.212.3:37151)
0924 11:35:09.794843 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:09.794850 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:09.794877 (+ 27us) server_negotiation.cc:407] Connection header received
0924 11:35:09.794945 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:09.794952 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:09.795170 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:09.795344 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:09.796801 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:09.798099 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:09.799519 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:09.800102 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:09.800208 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:09.802974 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:35:09.802984 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:35:09.802998 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:35:09.803038 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:35:09.805869 (+ 2831us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.212.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 11:35:09.806153 (+ 284us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36564: Request ticket server kudu/127.6.212.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":373,"threads_started":1}
W20260924 11:35:09.806576 16306 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36564: Request ticket server kudu/127.6.212.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 11:35:09.806667 6992 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 16126: Not authorized: Client connection negotiation failed: client connection to 127.6.212.3:37151: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.212.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 11:35:09.806849 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 16126
I20260924 11:35:09.823632 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 15775
I20260924 11:35:09.825434 15837 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:35:09.941246 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 15775
2026-09-24T11:35:09Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21235 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T11:35:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:35:10Z Disabled control of system clock
I20260924 11:35:10.024899 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:46763
--webserver_interface=127.6.212.62
--webserver_port=0
--builtin_ntp_servers=127.6.212.20:34413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:46763
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 11:35:10.361001 16311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:35:10.361286 16311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:35:10.361436 16311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:35:10.368825 16311 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:35:10.368937 16311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:35:10.369024 16311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:35:10.369132 16311 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:35:10.369218 16311 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:35:10.378540 16311 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:34413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:46763
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:46763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.212.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:35:10.379961 16311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:35:10.381771 16311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:35:10.393947 16316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:35:10.394016 16317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:35:10.394156 16319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:35:10.395087 16311 server_base.cc:1034] running on GCE node
I20260924 11:35:10.396044 16311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:35:10.397555 16311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:35:10.398787 16311 hybrid_clock.cc:648] HybridClock initialized: now 1790249710398751 us; error 27 us; skew 500 ppm
I20260924 11:35:10.399160 16311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:35:10.401600 16311 webserver.cc:460] Webserver started at http://127.6.212.62:37555/ using document root <none> and password file <none>
I20260924 11:35:10.402325 16311 fs_manager.cc:359] Metadata directory not provided
I20260924 11:35:10.402412 16311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:35:10.402772 16311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:35:10.405848 16311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "723061179aaf41ee8ec593ba0bf82ae1"
format_stamp: "Formatted at 2026-09-24 11:35:10 on dist-test-slave-cq7c"
server_key: "b9c685c2f49a846b54b59f2490a253e2"
server_key_iv: "2df24f5a68c134f0dd8b73292476c4bf"
server_key_version: "encryptionkey@0"
I20260924 11:35:10.406669 16311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "723061179aaf41ee8ec593ba0bf82ae1"
format_stamp: "Formatted at 2026-09-24 11:35:10 on dist-test-slave-cq7c"
server_key: "b9c685c2f49a846b54b59f2490a253e2"
server_key_iv: "2df24f5a68c134f0dd8b73292476c4bf"
server_key_version: "encryptionkey@0"
I20260924 11:35:10.412425 16311 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 11:35:10.416392 16325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:35:10.418028 16311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:35:10.418226 16311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "723061179aaf41ee8ec593ba0bf82ae1"
format_stamp: "Formatted at 2026-09-24 11:35:10 on dist-test-slave-cq7c"
server_key: "b9c685c2f49a846b54b59f2490a253e2"
server_key_iv: "2df24f5a68c134f0dd8b73292476c4bf"
server_key_version: "encryptionkey@0"
I20260924 11:35:10.418483 16311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:35:10.436210 16311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:35:10.437101 16311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:35:10.437407 16311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:35:10.457341 16311 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.62:46763
I20260924 11:35:10.457368 16376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.62:46763 every 8 connection(s)
I20260924 11:35:10.458746 16311 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:35:10.462568 16377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:35:10.468827 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 16311
I20260924 11:35:10.469045 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:35:10.469842 6992 external_mini_cluster.cc:1428] Setting key 93ecafe8deb0ae417e9fb50eba8879c8
I20260924 11:35:10.474177 16377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 723061179aaf41ee8ec593ba0bf82ae1: Bootstrap starting.
I20260924 11:35:10.477511 16377 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 723061179aaf41ee8ec593ba0bf82ae1: Neither blocks nor log segments found. Creating new log.
I20260924 11:35:10.478541 16377 log.cc:824] T 00000000000000000000000000000000 P 723061179aaf41ee8ec593ba0bf82ae1: Log is configured to *not* fsync() on all Append() calls
I20260924 11:35:10.481580 16377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 723061179aaf41ee8ec593ba0bf82ae1: No bootstrap required, opened a new log
I20260924 11:35:10.484383 16380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:10.472137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54510 (local address 127.6.212.62:46763)
0924 11:35:10.473190 (+ 1053us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:10.473201 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:10.473246 (+ 45us) server_negotiation.cc:407] Connection header received
0924 11:35:10.474942 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:10.474964 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:10.475488 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:10.475936 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:10.476595 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:10.477927 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:10.479313 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:10.480079 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:10.480286 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:10.481671 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:35:10.481703 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:35:10.481726 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:35:10.482417 (+ 691us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:35:10.482499 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:35:10.482521 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:35:10.482670 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 11:35:10.482921 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 11:35:10.483273 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":410,"threads_started":1}
I20260924 11:35:10.489800 16377 raft_consensus.cc:348] T 00000000000000000000000000000000 P 723061179aaf41ee8ec593ba0bf82ae1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723061179aaf41ee8ec593ba0bf82ae1" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 46763 } }
I20260924 11:35:10.490175 16377 raft_consensus.cc:374] T 00000000000000000000000000000000 P 723061179aaf41ee8ec593ba0bf82ae1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:35:10.490360 16377 raft_consensus.cc:729] T 00000000000000000000000000000000 P 723061179aaf41ee8ec593ba0bf82ae1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 723061179aaf41ee8ec593ba0bf82ae1, State: Initialized, Role: FOLLOWER
I20260924 11:35:10.491000 16377 consensus_queue.cc:260] T 00000000000000000000000000000000 P 723061179aaf41ee8ec593ba0bf82ae1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723061179aaf41ee8ec593ba0bf82ae1" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 46763 } }
I20260924 11:35:10.491252 16377 raft_consensus.cc:388] T 00000000000000000000000000000000 P 723061179aaf41ee8ec593ba0bf82ae1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:35:10.491423 16377 raft_consensus.cc:482] T 00000000000000000000000000000000 P 723061179aaf41ee8ec593ba0bf82ae1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:35:10.491605 16377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 723061179aaf41ee8ec593ba0bf82ae1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:35:10.493773 16377 raft_consensus.cc:504] T 00000000000000000000000000000000 P 723061179aaf41ee8ec593ba0bf82ae1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723061179aaf41ee8ec593ba0bf82ae1" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 46763 } }
I20260924 11:35:10.494262 16377 leader_election.cc:302] T 00000000000000000000000000000000 P 723061179aaf41ee8ec593ba0bf82ae1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 723061179aaf41ee8ec593ba0bf82ae1; no voters:
I20260924 11:35:10.494848 16377 leader_election.cc:288] T 00000000000000000000000000000000 P 723061179aaf41ee8ec593ba0bf82ae1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:35:10.495020 16382 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 723061179aaf41ee8ec593ba0bf82ae1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:35:10.496515 16382 raft_consensus.cc:686] T 00000000000000000000000000000000 P 723061179aaf41ee8ec593ba0bf82ae1 [term 1 LEADER]: Becoming Leader. State: Replica: 723061179aaf41ee8ec593ba0bf82ae1, State: Running, Role: LEADER
I20260924 11:35:10.497046 16377 sys_catalog.cc:564] T 00000000000000000000000000000000 P 723061179aaf41ee8ec593ba0bf82ae1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:35:10.497081 16382 consensus_queue.cc:237] T 00000000000000000000000000000000 P 723061179aaf41ee8ec593ba0bf82ae1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723061179aaf41ee8ec593ba0bf82ae1" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 46763 } }
I20260924 11:35:10.500144 16383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 723061179aaf41ee8ec593ba0bf82ae1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "723061179aaf41ee8ec593ba0bf82ae1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723061179aaf41ee8ec593ba0bf82ae1" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 46763 } } }
I20260924 11:35:10.500516 16383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 723061179aaf41ee8ec593ba0bf82ae1 [sys.catalog]: This master's current role is: LEADER
I20260924 11:35:10.500274 16384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 723061179aaf41ee8ec593ba0bf82ae1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 723061179aaf41ee8ec593ba0bf82ae1. Latest consensus state: current_term: 1 leader_uuid: "723061179aaf41ee8ec593ba0bf82ae1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723061179aaf41ee8ec593ba0bf82ae1" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 46763 } } }
I20260924 11:35:10.500653 16384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 723061179aaf41ee8ec593ba0bf82ae1 [sys.catalog]: This master's current role is: LEADER
I20260924 11:35:10.502291 16390 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:35:10.508414 16390 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:35:10.517696 16390 catalog_manager.cc:1347] Generated new cluster ID: fcce41d3e8ce4a1789fa7f51f6595e16
I20260924 11:35:10.517838 16390 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:35:10.542040 16390 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:35:10.543656 16390 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:35:10.555083 16390 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 723061179aaf41ee8ec593ba0bf82ae1: Generated new TSK 0
I20260924 11:35:10.556332 16390 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:35:10.568545 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.1:0
--local_ip_for_outbound_sockets=127.6.212.1
--webserver_interface=127.6.212.1
--webserver_port=0
--tserver_master_addrs=127.6.212.62:46763
--builtin_ntp_servers=127.6.212.20:34413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 11:35:10.910488 16401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:35:10.910861 16401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:35:10.911016 16401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:35:10.918907 16401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:35:10.919072 16401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:35:10.919258 16401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.1
I20260924 11:35:10.928982 16401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:34413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.212.1
--webserver_port=0
--tserver_master_addrs=127.6.212.62:46763
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.1
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:35:10.930339 16401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:35:10.932245 16401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:35:10.946643 16406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:35:10.946663 16409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:35:10.946774 16407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:35:10.947255 16401 server_base.cc:1034] running on GCE node
I20260924 11:35:10.948020 16401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:35:10.949054 16401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:35:10.950271 16401 hybrid_clock.cc:648] HybridClock initialized: now 1790249710950233 us; error 56 us; skew 500 ppm
I20260924 11:35:10.950618 16401 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:35:10.957110 16401 webserver.cc:460] Webserver started at http://127.6.212.1:35693/ using document root <none> and password file <none>
I20260924 11:35:10.957980 16401 fs_manager.cc:359] Metadata directory not provided
I20260924 11:35:10.958093 16401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:35:10.958523 16401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:35:10.961755 16401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/data/instance:
uuid: "bb536022a51e449aaebb5ab2551a754b"
format_stamp: "Formatted at 2026-09-24 11:35:10 on dist-test-slave-cq7c"
server_key: "578b9565c9c9a01697b5ad21ae6642dc"
server_key_iv: "f65b76d4e4e261dcdea124c1079f169f"
server_key_version: "encryptionkey@0"
I20260924 11:35:10.962620 16401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance:
uuid: "bb536022a51e449aaebb5ab2551a754b"
format_stamp: "Formatted at 2026-09-24 11:35:10 on dist-test-slave-cq7c"
server_key: "578b9565c9c9a01697b5ad21ae6642dc"
server_key_iv: "f65b76d4e4e261dcdea124c1079f169f"
server_key_version: "encryptionkey@0"
I20260924 11:35:10.968317 16401 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 11:35:10.971995 16415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:35:10.973409 16401 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:35:10.973605 16401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/wal
uuid: "bb536022a51e449aaebb5ab2551a754b"
format_stamp: "Formatted at 2026-09-24 11:35:10 on dist-test-slave-cq7c"
server_key: "578b9565c9c9a01697b5ad21ae6642dc"
server_key_iv: "f65b76d4e4e261dcdea124c1079f169f"
server_key_version: "encryptionkey@0"
I20260924 11:35:10.973837 16401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:35:10.989995 16401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:35:10.990898 16401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:35:10.991192 16401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:35:10.992269 16401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:35:10.993719 16401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:35:10.993844 16401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:35:10.993978 16401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:35:10.994069 16401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:35:11.031488 16401 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.1:40963
I20260924 11:35:11.031581 16527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.1:40963 every 8 connection(s)
I20260924 11:35:11.032989 16401 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
I20260924 11:35:11.036154 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 16401
I20260924 11:35:11.036379 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance
I20260924 11:35:11.037098 6992 external_mini_cluster.cc:1428] Setting key 7da1bf4fe3e38a3cbd9f870b844c68f6
I20260924 11:35:11.040287 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.2:0
--local_ip_for_outbound_sockets=127.6.212.2
--webserver_interface=127.6.212.2
--webserver_port=0
--tserver_master_addrs=127.6.212.62:46763
--builtin_ntp_servers=127.6.212.20:34413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 11:35:11.048292 16530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:11.035758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:42981 (local address 127.6.212.62:46763)
0924 11:35:11.036281 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:11.036290 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:11.037880 (+ 1590us) server_negotiation.cc:407] Connection header received
0924 11:35:11.040309 (+ 2429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:11.040317 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:11.040415 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:11.040614 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:11.041899 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:11.043129 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:11.044839 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:11.045620 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:11.045780 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:11.047153 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:35:11.047163 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:35:11.047171 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:35:11.047202 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:35:11.047261 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:35:11.047283 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:35:11.047435 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 11:35:11.047660 (+ 225us) server_negotiation.cc:299] Negotiation successful
0924 11:35:11.047779 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":236,"threads_started":1}
I20260924 11:35:11.049399 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:11.036531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:46763 (local address 127.6.212.1:42981)
0924 11:35:11.037699 (+ 1168us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:11.037747 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 11:35:11.039993 (+ 2246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:35:11.040924 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:35:11.040934 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:35:11.041040 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:35:11.041691 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:11.041718 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:11.043317 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:11.043321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:11.044679 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:11.044692 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:11.045847 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:11.045852 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:11.046055 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:11.046828 (+ 773us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:35:11.046830 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:35:11.046871 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:35:11.046951 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:35:11.047446 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:35:11.047453 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:35:11.047460 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 11:35:11.047612 (+ 152us) client_negotiation.cc:240] Negotiation successful
0924 11:35:11.047929 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":403,"threads_started":1}
I20260924 11:35:11.051016 16528 heartbeater.cc:347] Connected to a master server at 127.6.212.62:46763
I20260924 11:35:11.051342 16528 heartbeater.cc:464] Registering TS with master...
I20260924 11:35:11.052282 16528 heartbeater.cc:511] Master 127.6.212.62:46763 requested a full tablet report, sending...
I20260924 11:35:11.054623 16342 ts_manager.cc:194] Registered new tserver with Master: bb536022a51e449aaebb5ab2551a754b (127.6.212.1:40963)
I20260924 11:35:11.056237 16342 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.212.1:42981
W20260924 11:35:11.378120 16532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:35:11.378458 16532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:35:11.378644 16532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:35:11.386256 16532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:35:11.386425 16532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:35:11.386615 16532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.2
I20260924 11:35:11.396653 16532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:34413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.212.2
--webserver_port=0
--tserver_master_addrs=127.6.212.62:46763
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.2
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:35:11.398351 16532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:35:11.400216 16532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:35:11.413880 16537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:35:11.413949 16538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:35:11.414356 16532 server_base.cc:1034] running on GCE node
W20260924 11:35:11.413978 16540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:35:11.415248 16532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:35:11.416304 16532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:35:11.417548 16532 hybrid_clock.cc:648] HybridClock initialized: now 1790249711417470 us; error 63 us; skew 500 ppm
I20260924 11:35:11.417888 16532 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:35:11.420447 16532 webserver.cc:460] Webserver started at http://127.6.212.2:41563/ using document root <none> and password file <none>
I20260924 11:35:11.421175 16532 fs_manager.cc:359] Metadata directory not provided
I20260924 11:35:11.421324 16532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:35:11.421655 16532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:35:11.424669 16532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/data/instance:
uuid: "331dbefcec2b431388239e406334cf7e"
format_stamp: "Formatted at 2026-09-24 11:35:11 on dist-test-slave-cq7c"
server_key: "043ca29897a3a66cfe53dd722042852f"
server_key_iv: "0ac8ea473837016eba1bccb43bb679fa"
server_key_version: "encryptionkey@0"
I20260924 11:35:11.425469 16532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance:
uuid: "331dbefcec2b431388239e406334cf7e"
format_stamp: "Formatted at 2026-09-24 11:35:11 on dist-test-slave-cq7c"
server_key: "043ca29897a3a66cfe53dd722042852f"
server_key_iv: "0ac8ea473837016eba1bccb43bb679fa"
server_key_version: "encryptionkey@0"
I20260924 11:35:11.430493 16532 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 11:35:11.433885 16546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:35:11.435277 16532 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 11:35:11.435477 16532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/wal
uuid: "331dbefcec2b431388239e406334cf7e"
format_stamp: "Formatted at 2026-09-24 11:35:11 on dist-test-slave-cq7c"
server_key: "043ca29897a3a66cfe53dd722042852f"
server_key_iv: "0ac8ea473837016eba1bccb43bb679fa"
server_key_version: "encryptionkey@0"
I20260924 11:35:11.435693 16532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:35:11.450052 16532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:35:11.450984 16532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:35:11.451306 16532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:35:11.452366 16532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:35:11.454263 16532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:35:11.454371 16532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:35:11.454493 16532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:35:11.454574 16532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:35:11.490824 16532 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.2:38055
I20260924 11:35:11.490919 16658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.2:38055 every 8 connection(s)
I20260924 11:35:11.492233 16532 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
I20260924 11:35:11.497479 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 16532
I20260924 11:35:11.497677 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance
I20260924 11:35:11.498152 6992 external_mini_cluster.cc:1428] Setting key 2e1688b2bd898c46d479f7580a68af05
I20260924 11:35:11.501350 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.3:0
--local_ip_for_outbound_sockets=127.6.212.3
--webserver_interface=127.6.212.3
--webserver_port=0
--tserver_master_addrs=127.6.212.62:46763
--builtin_ntp_servers=127.6.212.20:34413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 11:35:11.506582 16530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:11.494806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:35553 (local address 127.6.212.62:46763)
0924 11:35:11.495015 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:11.495021 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:11.496480 (+ 1459us) server_negotiation.cc:407] Connection header received
0924 11:35:11.498402 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:11.498407 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:11.498493 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:11.498611 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:11.499888 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:11.500879 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:11.503634 (+ 2755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:11.504038 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:11.504175 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:11.505888 (+ 1713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:35:11.505892 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:35:11.505895 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:35:11.505914 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:35:11.505945 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:35:11.505957 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:35:11.506022 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 11:35:11.506289 (+ 267us) server_negotiation.cc:299] Negotiation successful
0924 11:35:11.506363 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 11:35:11.508186 16661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:11.495436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:46763 (local address 127.6.212.2:35553)
0924 11:35:11.496320 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:11.496363 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 11:35:11.498084 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:35:11.498880 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:35:11.498893 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:35:11.499019 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:35:11.499718 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:11.499737 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:11.501046 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:11.501051 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:11.503447 (+ 2396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:11.503467 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:11.504312 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:11.504321 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:11.504662 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:11.505472 (+ 810us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:35:11.505478 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:35:11.505536 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:35:11.505649 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:35:11.506049 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:35:11.506062 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:35:11.506074 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 11:35:11.506265 (+ 191us) client_negotiation.cc:240] Negotiation successful
0924 11:35:11.506703 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":255,"threads_started":1}
I20260924 11:35:11.509644 16659 heartbeater.cc:347] Connected to a master server at 127.6.212.62:46763
I20260924 11:35:11.509984 16659 heartbeater.cc:464] Registering TS with master...
I20260924 11:35:11.510735 16659 heartbeater.cc:511] Master 127.6.212.62:46763 requested a full tablet report, sending...
I20260924 11:35:11.512094 16342 ts_manager.cc:194] Registered new tserver with Master: 331dbefcec2b431388239e406334cf7e (127.6.212.2:38055)
I20260924 11:35:11.512885 16342 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.212.2:35553
W20260924 11:35:11.838950 16662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:35:11.839238 16662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:35:11.839454 16662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:35:11.846983 16662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:35:11.847092 16662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:35:11.847262 16662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.3
I20260924 11:35:11.856863 16662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:34413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.212.3
--webserver_port=0
--tserver_master_addrs=127.6.212.62:46763
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.3
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:35:11.858206 16662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:35:11.860021 16662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:35:11.873656 16668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:35:11.873618 16670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:35:11.873708 16667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:35:11.874698 16662 server_base.cc:1034] running on GCE node
I20260924 11:35:11.875329 16662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:35:11.876411 16662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:35:11.877727 16662 hybrid_clock.cc:648] HybridClock initialized: now 1790249711877651 us; error 68 us; skew 500 ppm
I20260924 11:35:11.878150 16662 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:35:11.880791 16662 webserver.cc:460] Webserver started at http://127.6.212.3:37275/ using document root <none> and password file <none>
I20260924 11:35:11.881675 16662 fs_manager.cc:359] Metadata directory not provided
I20260924 11:35:11.881810 16662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:35:11.882215 16662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:35:11.885426 16662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/data/instance:
uuid: "901a9a2855a648da80be93671932fff5"
format_stamp: "Formatted at 2026-09-24 11:35:11 on dist-test-slave-cq7c"
server_key: "114b3abbbfd133bca6b404d88ce3f024"
server_key_iv: "9b3ef36e217a01ec5e67d0a5aa2db807"
server_key_version: "encryptionkey@0"
I20260924 11:35:11.886292 16662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance:
uuid: "901a9a2855a648da80be93671932fff5"
format_stamp: "Formatted at 2026-09-24 11:35:11 on dist-test-slave-cq7c"
server_key: "114b3abbbfd133bca6b404d88ce3f024"
server_key_iv: "9b3ef36e217a01ec5e67d0a5aa2db807"
server_key_version: "encryptionkey@0"
I20260924 11:35:11.891924 16662 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260924 11:35:11.895552 16676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:35:11.897043 16662 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260924 11:35:11.897256 16662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/wal
uuid: "901a9a2855a648da80be93671932fff5"
format_stamp: "Formatted at 2026-09-24 11:35:11 on dist-test-slave-cq7c"
server_key: "114b3abbbfd133bca6b404d88ce3f024"
server_key_iv: "9b3ef36e217a01ec5e67d0a5aa2db807"
server_key_version: "encryptionkey@0"
I20260924 11:35:11.897490 16662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:35:11.912626 16662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:35:11.913551 16662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:35:11.913867 16662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:35:11.914914 16662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:35:11.916592 16662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:35:11.916687 16662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:35:11.916860 16662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:35:11.916965 16662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:35:11.952090 16662 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.3:36885
I20260924 11:35:11.952190 16788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.3:36885 every 8 connection(s)
I20260924 11:35:11.953397 16662 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
I20260924 11:35:11.957885 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 16662
I20260924 11:35:11.958086 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance
I20260924 11:35:11.958644 6992 external_mini_cluster.cc:1428] Setting key 3b61109195fb19968c9e2ef2a6c9da0e
I20260924 11:35:11.966281 16530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:11.956002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:36719 (local address 127.6.212.62:46763)
0924 11:35:11.956213 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:11.956220 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:11.957629 (+ 1409us) server_negotiation.cc:407] Connection header received
0924 11:35:11.960066 (+ 2437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:11.960071 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:11.960158 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:11.960275 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:11.961518 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:11.962302 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:11.963872 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:11.964276 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:11.964371 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:11.965611 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:35:11.965615 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:35:11.965619 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:35:11.965635 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:35:11.965665 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:35:11.965675 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:35:11.965745 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 11:35:11.965968 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 11:35:11.966040 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 11:35:11.967319 16791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:11.956569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:46763 (local address 127.6.212.3:36719)
0924 11:35:11.957431 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:11.957468 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 11:35:11.959764 (+ 2296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:35:11.960521 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:35:11.960533 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:35:11.960667 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:35:11.961321 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:11.961337 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:11.962426 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:11.962430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:11.963749 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:11.963760 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:11.964449 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:11.964454 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:11.964647 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:11.965302 (+ 655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:35:11.965304 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:35:11.965351 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:35:11.965426 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:35:11.965759 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:35:11.965767 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:35:11.965773 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 11:35:11.965931 (+ 158us) client_negotiation.cc:240] Negotiation successful
0924 11:35:11.966216 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":261,"threads_started":1}
I20260924 11:35:11.968772 16789 heartbeater.cc:347] Connected to a master server at 127.6.212.62:46763
I20260924 11:35:11.969084 16789 heartbeater.cc:464] Registering TS with master...
I20260924 11:35:11.969938 16789 heartbeater.cc:511] Master 127.6.212.62:46763 requested a full tablet report, sending...
I20260924 11:35:11.971268 16342 ts_manager.cc:194] Registered new tserver with Master: 901a9a2855a648da80be93671932fff5 (127.6.212.3:36885)
I20260924 11:35:11.972126 16342 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.212.3:36719
I20260924 11:35:11.974000 6992 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 11:35:11.980094 16530 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:35:11.978444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54512 (local address 127.6.212.62:46763)
0924 11:35:11.978628 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:11.978635 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:11.979065 (+ 430us) server_negotiation.cc:407] Connection header received
0924 11:35:11.979253 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:11.979258 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:11.979335 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:11.979465 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:11.979854 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54512: BlockingRecv error: recv got EOF from 127.0.0.1:54512 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 11:35:11.981092 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 16401
I20260924 11:35:11.990689 16798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:11.981984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39574 (local address 127.6.212.1:40963)
0924 11:35:11.982456 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:11.982463 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:11.982489 (+ 26us) server_negotiation.cc:407] Connection header received
0924 11:35:11.982546 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:11.982552 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:11.982767 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:11.982897 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:11.983468 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:11.984637 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:11.986675 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:11.987477 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:11.987577 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:11.989070 (+ 1493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:35:11.989100 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:35:11.989123 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:35:11.989787 (+ 664us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:35:11.989842 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:35:11.989865 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:35:11.989965 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 11:35:11.990174 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 11:35:11.990384 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":254,"threads_started":1}
I20260924 11:35:11.992086 16523 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:35:12.072544 16528 heartbeater.cc:503] Master 127.6.212.62:46763 was elected leader, sending a full tablet report...
I20260924 11:35:12.073769 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 16401
W20260924 11:35:12.091722 16329 connection.cc:535] server connection from 127.6.212.1:42981 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 11:35:12.092044 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 16532
I20260924 11:35:12.100538 16800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:12.092988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43630 (local address 127.6.212.2:38055)
0924 11:35:12.093561 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:12.093569 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:12.093597 (+ 28us) server_negotiation.cc:407] Connection header received
0924 11:35:12.093729 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:12.093736 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:12.093970 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:12.094169 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:12.094597 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:12.095803 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:12.097188 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:12.097805 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:12.097962 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:12.099052 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:35:12.099061 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:35:12.099076 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:35:12.099801 (+ 725us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:35:12.099850 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:35:12.099876 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:35:12.099971 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 11:35:12.100124 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 11:35:12.100251 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":292,"threads_started":1}
I20260924 11:35:12.101897 16654 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:35:12.184860 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 16532
I20260924 11:35:12.207131 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 16662
I20260924 11:35:12.217104 16802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:12.208177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53782 (local address 127.6.212.3:36885)
0924 11:35:12.208750 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:12.208757 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:12.208789 (+ 32us) server_negotiation.cc:407] Connection header received
0924 11:35:12.208846 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:12.208853 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:12.209072 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:12.209249 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:12.209732 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:12.210757 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:12.213006 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:12.213667 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:12.213790 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:12.215245 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:35:12.215254 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:35:12.215269 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:35:12.216091 (+ 822us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:35:12.216146 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:35:12.216167 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:35:12.216276 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 11:35:12.216492 (+ 216us) server_negotiation.cc:299] Negotiation successful
0924 11:35:12.216611 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":253,"threads_started":1}
I20260924 11:35:12.218763 16784 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:35:12.304041 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 16662
I20260924 11:35:12.319996 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 16311
I20260924 11:35:12.321193 16372 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:35:12.427551 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 16311
2026-09-24T11:35:12Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2480 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T11:35:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:35:12Z Disabled control of system clock
I20260924 11:35:12.506722 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:38767
--webserver_interface=127.6.212.62
--webserver_port=0
--builtin_ntp_servers=127.6.212.20:32825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:38767
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 11:35:12.843984 16808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:35:12.844300 16808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:35:12.844441 16808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:35:12.851828 16808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:35:12.851944 16808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:35:12.852005 16808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:35:12.852136 16808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:35:12.852190 16808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:35:12.861863 16808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:32825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:38767
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:38767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.212.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:35:12.863281 16808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:35:12.865196 16808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:35:12.877095 16816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:35:12.877136 16813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:35:12.877163 16814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:35:12.878297 16808 server_base.cc:1034] running on GCE node
I20260924 11:35:12.879159 16808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:35:12.880702 16808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:35:12.881958 16808 hybrid_clock.cc:648] HybridClock initialized: now 1790249712881904 us; error 49 us; skew 500 ppm
I20260924 11:35:12.882355 16808 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:35:12.884938 16808 webserver.cc:460] Webserver started at http://127.6.212.62:34845/ using document root <none> and password file <none>
I20260924 11:35:12.885648 16808 fs_manager.cc:359] Metadata directory not provided
I20260924 11:35:12.885793 16808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:35:12.886117 16808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:35:12.890754 16808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "aa70e596555a4648859cfbec6d63885f"
format_stamp: "Formatted at 2026-09-24 11:35:12 on dist-test-slave-cq7c"
server_key: "26ae8059c70ca00b7029eb3efd7ded67"
server_key_iv: "97269c676d5f26803977174f3b138c37"
server_key_version: "encryptionkey@0"
I20260924 11:35:12.891641 16808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "aa70e596555a4648859cfbec6d63885f"
format_stamp: "Formatted at 2026-09-24 11:35:12 on dist-test-slave-cq7c"
server_key: "26ae8059c70ca00b7029eb3efd7ded67"
server_key_iv: "97269c676d5f26803977174f3b138c37"
server_key_version: "encryptionkey@0"
I20260924 11:35:12.896909 16808 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 11:35:12.900719 16822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:35:12.902146 16808 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:35:12.902386 16808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "aa70e596555a4648859cfbec6d63885f"
format_stamp: "Formatted at 2026-09-24 11:35:12 on dist-test-slave-cq7c"
server_key: "26ae8059c70ca00b7029eb3efd7ded67"
server_key_iv: "97269c676d5f26803977174f3b138c37"
server_key_version: "encryptionkey@0"
I20260924 11:35:12.902616 16808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:35:12.908450 16808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:35:12.909236 16808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:35:12.909535 16808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:35:12.928681 16808 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.62:38767
I20260924 11:35:12.928778 16873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.62:38767 every 8 connection(s)
I20260924 11:35:12.930023 16808 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:35:12.931469 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 16808
I20260924 11:35:12.931667 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:35:12.932536 6992 external_mini_cluster.cc:1428] Setting key 0c84aa73ed268a215a03c114d757c74d
I20260924 11:35:12.935245 16874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:35:12.941469 16877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:12.934737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52720 (local address 127.6.212.62:38767)
0924 11:35:12.935853 (+ 1116us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:12.935864 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:12.935905 (+ 41us) server_negotiation.cc:407] Connection header received
0924 11:35:12.937519 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:12.937542 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:12.938020 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:12.938457 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:12.939630 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:35:12.939640 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:35:12.939653 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:35:12.939668 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:35:12.939757 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:35:12.939778 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:35:12.940035 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0924 11:35:12.940181 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 11:35:12.940466 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":380,"threads_started":1}
I20260924 11:35:12.949115 16874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa70e596555a4648859cfbec6d63885f: Bootstrap starting.
I20260924 11:35:12.952620 16874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa70e596555a4648859cfbec6d63885f: Neither blocks nor log segments found. Creating new log.
I20260924 11:35:12.953861 16874 log.cc:824] T 00000000000000000000000000000000 P aa70e596555a4648859cfbec6d63885f: Log is configured to *not* fsync() on all Append() calls
I20260924 11:35:12.957214 16874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa70e596555a4648859cfbec6d63885f: No bootstrap required, opened a new log
I20260924 11:35:12.965533 16874 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa70e596555a4648859cfbec6d63885f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa70e596555a4648859cfbec6d63885f" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 38767 } }
I20260924 11:35:12.965929 16874 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa70e596555a4648859cfbec6d63885f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:35:12.966030 16874 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa70e596555a4648859cfbec6d63885f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa70e596555a4648859cfbec6d63885f, State: Initialized, Role: FOLLOWER
I20260924 11:35:12.966640 16874 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa70e596555a4648859cfbec6d63885f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa70e596555a4648859cfbec6d63885f" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 38767 } }
I20260924 11:35:12.966866 16874 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa70e596555a4648859cfbec6d63885f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:35:12.966995 16874 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa70e596555a4648859cfbec6d63885f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:35:12.967178 16874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa70e596555a4648859cfbec6d63885f [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:35:12.969647 16874 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa70e596555a4648859cfbec6d63885f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa70e596555a4648859cfbec6d63885f" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 38767 } }
I20260924 11:35:12.970149 16874 leader_election.cc:302] T 00000000000000000000000000000000 P aa70e596555a4648859cfbec6d63885f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa70e596555a4648859cfbec6d63885f; no voters:
I20260924 11:35:12.970749 16874 leader_election.cc:288] T 00000000000000000000000000000000 P aa70e596555a4648859cfbec6d63885f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:35:12.970904 16879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa70e596555a4648859cfbec6d63885f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:35:12.971642 16879 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa70e596555a4648859cfbec6d63885f [term 1 LEADER]: Becoming Leader. State: Replica: aa70e596555a4648859cfbec6d63885f, State: Running, Role: LEADER
I20260924 11:35:12.972244 16879 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa70e596555a4648859cfbec6d63885f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa70e596555a4648859cfbec6d63885f" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 38767 } }
I20260924 11:35:12.972860 16874 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa70e596555a4648859cfbec6d63885f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:35:12.975016 16880 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa70e596555a4648859cfbec6d63885f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa70e596555a4648859cfbec6d63885f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa70e596555a4648859cfbec6d63885f" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 38767 } } }
I20260924 11:35:12.975399 16880 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa70e596555a4648859cfbec6d63885f [sys.catalog]: This master's current role is: LEADER
I20260924 11:35:12.976270 16886 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:35:12.976112 16881 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa70e596555a4648859cfbec6d63885f [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa70e596555a4648859cfbec6d63885f. Latest consensus state: current_term: 1 leader_uuid: "aa70e596555a4648859cfbec6d63885f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa70e596555a4648859cfbec6d63885f" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 38767 } } }
I20260924 11:35:12.976464 16881 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa70e596555a4648859cfbec6d63885f [sys.catalog]: This master's current role is: LEADER
I20260924 11:35:12.981397 16886 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:35:12.990357 16886 catalog_manager.cc:1347] Generated new cluster ID: ed74137c814448fa826e86329d9213c2
I20260924 11:35:12.990500 16886 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:35:13.016877 16886 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:35:13.017364 16886 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:35:13.032009 16886 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa70e596555a4648859cfbec6d63885f: Generated new TSK 0
I20260924 11:35:13.032910 16886 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:35:13.040900 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.1:0
--local_ip_for_outbound_sockets=127.6.212.1
--webserver_interface=127.6.212.1
--webserver_port=0
--tserver_master_addrs=127.6.212.62:38767
--builtin_ntp_servers=127.6.212.20:32825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 11:35:13.381752 16898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:35:13.382045 16898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:35:13.382196 16898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:35:13.389575 16898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:35:13.389685 16898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:35:13.389859 16898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.1
I20260924 11:35:13.399135 16898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:32825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.212.1
--webserver_port=0
--tserver_master_addrs=127.6.212.62:38767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.1
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:35:13.400525 16898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:35:13.402441 16898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:35:13.416384 16906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:35:13.416453 16904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:35:13.416419 16903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:35:13.417096 16898 server_base.cc:1034] running on GCE node
I20260924 11:35:13.417766 16898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:35:13.418754 16898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:35:13.419955 16898 hybrid_clock.cc:648] HybridClock initialized: now 1790249713419887 us; error 62 us; skew 500 ppm
I20260924 11:35:13.420317 16898 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:35:13.422792 16898 webserver.cc:460] Webserver started at http://127.6.212.1:43623/ using document root <none> and password file <none>
I20260924 11:35:13.423656 16898 fs_manager.cc:359] Metadata directory not provided
I20260924 11:35:13.423779 16898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:35:13.424134 16898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:35:13.427027 16898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/data/instance:
uuid: "2e19c9ae667e40e8b590dc36175ec745"
format_stamp: "Formatted at 2026-09-24 11:35:13 on dist-test-slave-cq7c"
server_key: "d3ae24b1af4d774f74dcb56ff2618f6e"
server_key_iv: "0b97a90ac68726dfcf8a109cf777c96e"
server_key_version: "encryptionkey@0"
I20260924 11:35:13.427851 16898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance:
uuid: "2e19c9ae667e40e8b590dc36175ec745"
format_stamp: "Formatted at 2026-09-24 11:35:13 on dist-test-slave-cq7c"
server_key: "d3ae24b1af4d774f74dcb56ff2618f6e"
server_key_iv: "0b97a90ac68726dfcf8a109cf777c96e"
server_key_version: "encryptionkey@0"
I20260924 11:35:13.432870 16898 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 11:35:13.436396 16912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:35:13.438019 16898 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 11:35:13.438235 16898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/wal
uuid: "2e19c9ae667e40e8b590dc36175ec745"
format_stamp: "Formatted at 2026-09-24 11:35:13 on dist-test-slave-cq7c"
server_key: "d3ae24b1af4d774f74dcb56ff2618f6e"
server_key_iv: "0b97a90ac68726dfcf8a109cf777c96e"
server_key_version: "encryptionkey@0"
I20260924 11:35:13.438479 16898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:35:13.444877 16898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:35:13.445783 16898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:35:13.446082 16898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:35:13.447046 16898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:35:13.448690 16898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:35:13.448820 16898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:35:13.448967 16898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:35:13.449064 16898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:35:13.487905 16898 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.1:39519
I20260924 11:35:13.487967 17024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.1:39519 every 8 connection(s)
I20260924 11:35:13.489375 16898 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
I20260924 11:35:13.495571 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 16898
I20260924 11:35:13.495779 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance
I20260924 11:35:13.496388 6992 external_mini_cluster.cc:1428] Setting key f9840e9b85675d655ef69f45d84ba544
I20260924 11:35:13.498602 17027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:13.492419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:38993 (local address 127.6.212.62:38767)
0924 11:35:13.492870 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:13.492874 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:13.494181 (+ 1307us) server_negotiation.cc:407] Connection header received
0924 11:35:13.496079 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:13.496084 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:13.496170 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:13.496292 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:13.497836 (+ 1544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:35:13.497842 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:35:13.497847 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:35:13.497872 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:35:13.497912 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:35:13.497925 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:35:13.498060 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 11:35:13.498254 (+ 194us) server_negotiation.cc:299] Negotiation successful
0924 11:35:13.498318 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":225,"threads_started":1}
I20260924 11:35:13.499297 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.2:0
--local_ip_for_outbound_sockets=127.6.212.2
--webserver_interface=127.6.212.2
--webserver_port=0
--tserver_master_addrs=127.6.212.62:38767
--builtin_ntp_servers=127.6.212.20:32825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 11:35:13.499737 17028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:13.493047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:38767 (local address 127.6.212.1:38993)
0924 11:35:13.494006 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:13.494059 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 11:35:13.495742 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:35:13.496570 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:35:13.496579 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:35:13.496683 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:35:13.497467 (+ 784us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:35:13.497469 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:35:13.497510 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:35:13.497604 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:35:13.498023 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:35:13.498034 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:35:13.498044 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 11:35:13.498346 (+ 302us) client_negotiation.cc:240] Negotiation successful
0924 11:35:13.498634 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":286,"threads_started":1}
I20260924 11:35:13.504796 17025 heartbeater.cc:347] Connected to a master server at 127.6.212.62:38767
I20260924 11:35:13.505137 17025 heartbeater.cc:464] Registering TS with master...
I20260924 11:35:13.505821 17025 heartbeater.cc:511] Master 127.6.212.62:38767 requested a full tablet report, sending...
I20260924 11:35:13.508122 16839 ts_manager.cc:194] Registered new tserver with Master: 2e19c9ae667e40e8b590dc36175ec745 (127.6.212.1:39519)
W20260924 11:35:13.849498 17029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:35:13.849871 17029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:35:13.850057 17029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:35:13.858325 17029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:35:13.858489 17029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:35:13.858693 17029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.2
I20260924 11:35:13.868662 17029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:32825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.212.2
--webserver_port=0
--tserver_master_addrs=127.6.212.62:38767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.2
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:35:13.870043 17029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:35:13.871943 17029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:35:13.885831 17037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:35:13.885867 17035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:35:13.885800 17034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:35:13.886698 17029 server_base.cc:1034] running on GCE node
I20260924 11:35:13.887444 17029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:35:13.888463 17029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:35:13.889855 17029 hybrid_clock.cc:648] HybridClock initialized: now 1790249713889763 us; error 80 us; skew 500 ppm
I20260924 11:35:13.890254 17029 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:35:13.892725 17029 webserver.cc:460] Webserver started at http://127.6.212.2:38207/ using document root <none> and password file <none>
I20260924 11:35:13.893440 17029 fs_manager.cc:359] Metadata directory not provided
I20260924 11:35:13.893579 17029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:35:13.893959 17029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:35:13.896896 17029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/data/instance:
uuid: "bd5a9b6a2fa048078b33b15de1b481cc"
format_stamp: "Formatted at 2026-09-24 11:35:13 on dist-test-slave-cq7c"
server_key: "0589cd6b60f668c3a60e0d7fa823b36d"
server_key_iv: "a17ac95548f233be394798cc7c3ac8f4"
server_key_version: "encryptionkey@0"
I20260924 11:35:13.897667 17029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance:
uuid: "bd5a9b6a2fa048078b33b15de1b481cc"
format_stamp: "Formatted at 2026-09-24 11:35:13 on dist-test-slave-cq7c"
server_key: "0589cd6b60f668c3a60e0d7fa823b36d"
server_key_iv: "a17ac95548f233be394798cc7c3ac8f4"
server_key_version: "encryptionkey@0"
I20260924 11:35:13.902701 17029 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 11:35:13.906466 17043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:35:13.907974 17029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:35:13.908143 17029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/wal
uuid: "bd5a9b6a2fa048078b33b15de1b481cc"
format_stamp: "Formatted at 2026-09-24 11:35:13 on dist-test-slave-cq7c"
server_key: "0589cd6b60f668c3a60e0d7fa823b36d"
server_key_iv: "a17ac95548f233be394798cc7c3ac8f4"
server_key_version: "encryptionkey@0"
I20260924 11:35:13.908375 17029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:35:13.914067 17029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:35:13.914855 17029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:35:13.915117 17029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:35:13.916071 17029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:35:13.917435 17029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:35:13.917557 17029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:35:13.917692 17029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:35:13.917784 17029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:35:13.953716 17029 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.2:34433
I20260924 11:35:13.953801 17155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.2:34433 every 8 connection(s)
I20260924 11:35:13.954990 17029 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
I20260924 11:35:13.955489 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 17029
I20260924 11:35:13.955905 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance
I20260924 11:35:13.956475 6992 external_mini_cluster.cc:1428] Setting key 2fa3e7414adc42e98c24275582099947
I20260924 11:35:13.959530 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.3:0
--local_ip_for_outbound_sockets=127.6.212.3
--webserver_interface=127.6.212.3
--webserver_port=0
--tserver_master_addrs=127.6.212.62:38767
--builtin_ntp_servers=127.6.212.20:32825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 11:35:13.963867 17027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:13.957929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:44035 (local address 127.6.212.62:38767)
0924 11:35:13.958172 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:13.958178 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:13.959708 (+ 1530us) server_negotiation.cc:407] Connection header received
0924 11:35:13.961423 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:13.961431 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:13.961506 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:13.961614 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:13.962995 (+ 1381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:35:13.963003 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:35:13.963010 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:35:13.963033 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:35:13.963080 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:35:13.963098 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:35:13.963226 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 11:35:13.963472 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 11:35:13.963536 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 11:35:13.964841 17158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:13.958460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:38767 (local address 127.6.212.2:44035)
0924 11:35:13.959567 (+ 1107us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:13.959612 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 11:35:13.961148 (+ 1536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:35:13.961841 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:35:13.961850 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:35:13.961941 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:35:13.962685 (+ 744us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:35:13.962688 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:35:13.962727 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:35:13.962822 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:35:13.963209 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:35:13.963217 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:35:13.963222 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 11:35:13.963574 (+ 352us) client_negotiation.cc:240] Negotiation successful
0924 11:35:13.963816 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":332,"threads_started":1}
I20260924 11:35:13.966274 17156 heartbeater.cc:347] Connected to a master server at 127.6.212.62:38767
I20260924 11:35:13.966586 17156 heartbeater.cc:464] Registering TS with master...
I20260924 11:35:13.967160 17156 heartbeater.cc:511] Master 127.6.212.62:38767 requested a full tablet report, sending...
I20260924 11:35:13.968595 16839 ts_manager.cc:194] Registered new tserver with Master: bd5a9b6a2fa048078b33b15de1b481cc (127.6.212.2:34433)
W20260924 11:35:14.313824 17159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:35:14.314227 17159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:35:14.314415 17159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:35:14.322652 17159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:35:14.322822 17159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:35:14.323019 17159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.3
I20260924 11:35:14.332495 17159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:32825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.212.3
--webserver_port=0
--tserver_master_addrs=127.6.212.62:38767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.3
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:35:14.333940 17159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:35:14.335870 17159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:35:14.350378 17164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:35:14.350389 17165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:35:14.350492 17167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:35:14.351416 17159 server_base.cc:1034] running on GCE node
I20260924 11:35:14.352306 17159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:35:14.353405 17159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:35:14.354714 17159 hybrid_clock.cc:648] HybridClock initialized: now 1790249714354654 us; error 52 us; skew 500 ppm
I20260924 11:35:14.355067 17159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:35:14.357767 17159 webserver.cc:460] Webserver started at http://127.6.212.3:35599/ using document root <none> and password file <none>
I20260924 11:35:14.358479 17159 fs_manager.cc:359] Metadata directory not provided
I20260924 11:35:14.358616 17159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:35:14.358951 17159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:35:14.361934 17159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/data/instance:
uuid: "753f7a71e4354cd4a0a76cbca5adfea3"
format_stamp: "Formatted at 2026-09-24 11:35:14 on dist-test-slave-cq7c"
server_key: "d4a7283a984b4a47d8c6f26a771f386b"
server_key_iv: "3c5652366647d2bf1bfbaf2b8dd6580d"
server_key_version: "encryptionkey@0"
I20260924 11:35:14.362738 17159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance:
uuid: "753f7a71e4354cd4a0a76cbca5adfea3"
format_stamp: "Formatted at 2026-09-24 11:35:14 on dist-test-slave-cq7c"
server_key: "d4a7283a984b4a47d8c6f26a771f386b"
server_key_iv: "3c5652366647d2bf1bfbaf2b8dd6580d"
server_key_version: "encryptionkey@0"
I20260924 11:35:14.368403 17159 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260924 11:35:14.372793 17173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:35:14.374552 17159 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 11:35:14.374754 17159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/wal
uuid: "753f7a71e4354cd4a0a76cbca5adfea3"
format_stamp: "Formatted at 2026-09-24 11:35:14 on dist-test-slave-cq7c"
server_key: "d4a7283a984b4a47d8c6f26a771f386b"
server_key_iv: "3c5652366647d2bf1bfbaf2b8dd6580d"
server_key_version: "encryptionkey@0"
I20260924 11:35:14.374994 17159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:35:14.381059 17159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:35:14.381965 17159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:35:14.382259 17159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:35:14.383268 17159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:35:14.384876 17159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:35:14.385001 17159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:35:14.385144 17159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:35:14.385236 17159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:35:14.418951 17159 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.3:40553
I20260924 11:35:14.419013 17285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.3:40553 every 8 connection(s)
I20260924 11:35:14.420363 17159 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
I20260924 11:35:14.425113 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 17159
I20260924 11:35:14.425300 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance
I20260924 11:35:14.425769 6992 external_mini_cluster.cc:1428] Setting key fe8d0210b261606df2ecd8405d351241
I20260924 11:35:14.429242 17027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:14.423185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:45877 (local address 127.6.212.62:38767)
0924 11:35:14.423401 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:14.423408 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:14.424936 (+ 1528us) server_negotiation.cc:407] Connection header received
0924 11:35:14.426899 (+ 1963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:14.426905 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:14.426989 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:14.427096 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:14.428513 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:35:14.428519 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:35:14.428523 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:35:14.428542 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:35:14.428582 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:35:14.428594 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:35:14.428719 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 11:35:14.428906 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 11:35:14.428972 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 11:35:14.430234 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:14.423802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:38767 (local address 127.6.212.3:45877)
0924 11:35:14.424768 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:14.424805 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 11:35:14.426632 (+ 1827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:35:14.427341 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:35:14.427350 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:35:14.427512 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:35:14.428212 (+ 700us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:35:14.428214 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:35:14.428255 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:35:14.428343 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:35:14.428687 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:35:14.428697 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:35:14.428705 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 11:35:14.429001 (+ 296us) client_negotiation.cc:240] Negotiation successful
0924 11:35:14.429294 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":307,"threads_started":1}
I20260924 11:35:14.431684 17286 heartbeater.cc:347] Connected to a master server at 127.6.212.62:38767
I20260924 11:35:14.431969 17286 heartbeater.cc:464] Registering TS with master...
I20260924 11:35:14.432504 17286 heartbeater.cc:511] Master 127.6.212.62:38767 requested a full tablet report, sending...
I20260924 11:35:14.433982 16839 ts_manager.cc:194] Registered new tserver with Master: 753f7a71e4354cd4a0a76cbca5adfea3 (127.6.212.3:40553)
I20260924 11:35:14.441553 6992 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 11:35:14.448685 17027 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:35:14.446606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52724 (local address 127.6.212.62:38767)
0924 11:35:14.446909 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:14.446920 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:14.447432 (+ 512us) server_negotiation.cc:407] Connection header received
0924 11:35:14.447680 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:14.447686 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:14.447758 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:14.447877 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:14.448280 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52724: BlockingRecv error: recv got EOF from 127.0.0.1:52724 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
I20260924 11:35:14.449164 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 16898
I20260924 11:35:14.453508 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:14.450466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35806 (local address 127.6.212.1:39519)
0924 11:35:14.450994 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:14.451002 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:14.451029 (+ 27us) server_negotiation.cc:407] Connection header received
0924 11:35:14.451098 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:14.451104 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:14.451344 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:14.451605 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:14.452701 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:35:14.452707 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:35:14.452720 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:35:14.452737 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:35:14.452779 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:35:14.452793 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:35:14.452949 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 11:35:14.453076 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 11:35:14.453184 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":215,"threads_started":1}
I20260924 11:35:14.454945 17020 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:35:14.537185 17025 heartbeater.cc:503] Master 127.6.212.62:38767 was elected leader, sending a full tablet report...
I20260924 11:35:14.538618 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 16898
W20260924 11:35:14.552506 16823 connection.cc:535] server connection from 127.6.212.1:38993 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 11:35:14.552888 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 17029
I20260924 11:35:14.556743 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:14.553839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43882 (local address 127.6.212.2:34433)
0924 11:35:14.554343 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:14.554351 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:14.554379 (+ 28us) server_negotiation.cc:407] Connection header received
0924 11:35:14.554444 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:14.554451 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:14.554743 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:14.554926 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:14.555990 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:35:14.555996 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:35:14.556008 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:35:14.556025 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:35:14.556065 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:35:14.556079 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:35:14.556228 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 11:35:14.556354 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 11:35:14.556456 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":234,"threads_started":1}
I20260924 11:35:14.558166 17151 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:35:14.640084 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 17029
I20260924 11:35:14.656364 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 17159
I20260924 11:35:14.660336 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:14.657329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38842 (local address 127.6.212.3:40553)
0924 11:35:14.657969 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:14.657977 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:14.658007 (+ 30us) server_negotiation.cc:407] Connection header received
0924 11:35:14.658065 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:14.658072 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:14.658292 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:14.658417 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:14.659605 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:35:14.659612 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:35:14.659624 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:35:14.659635 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:35:14.659674 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:35:14.659687 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:35:14.659825 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 11:35:14.659880 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 11:35:14.659987 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":299,"threads_started":1}
I20260924 11:35:14.661588 17281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:35:14.743917 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 17159
I20260924 11:35:14.759848 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 16808
I20260924 11:35:14.761082 16869 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:35:14.865063 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 16808
2026-09-24T11:35:14Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2436 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:35:14 dist-test-slave-cq7c krb5kdc[17304](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:35:14 dist-test-slave-cq7c krb5kdc[17304](info): set up 2 sockets
Sep 24 11:35:14 dist-test-slave-cq7c krb5kdc[17304](info): commencing operation
krb5kdc: starting...
W20260924 11:35:15.968449 6992 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.007s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Sep 24 11:35:16 dist-test-slave-cq7c krb5kdc[17304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:35:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:35:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.212.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
I20260924 11:35:16.233453 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:46847
--webserver_interface=127.6.212.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:46847
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:35:16.590945 17317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:35:16.591293 17317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:35:16.591480 17317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:35:16.599768 17317 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:35:16.599893 17317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:35:16.599943 17317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:35:16.600006 17317 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:35:16.600057 17317 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:35:16.611433 17317 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:46847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:46847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.212.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:35:16.612841 17317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:35:16.614207 17317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:35:16.627826 17322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:35:16.627804 17323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:35:16.627804 17325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:35:16.629065 17317 server_base.cc:1034] running on GCE node
I20260924 11:35:16.629980 17317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:35:16.631816 17317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:35:16.633114 17317 hybrid_clock.cc:648] HybridClock initialized: now 1790249716633069 us; error 55 us; skew 500 ppm
I20260924 11:35:16.633522 17317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:35:16 dist-test-slave-cq7c krb5kdc[17304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249716, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:35:16.641397 17317 init.cc:377] Logged in from keytab as kudu/127.6.212.62@KRBTEST.COM (short username kudu)
I20260924 11:35:16.643175 17317 webserver.cc:460] Webserver started at http://127.6.212.62:43583/ using document root <none> and password file <none>
I20260924 11:35:16.644032 17317 fs_manager.cc:359] Metadata directory not provided
I20260924 11:35:16.644184 17317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:35:16.644500 17317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:35:16.647672 17317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "a23347ea14794696874f43624ef61cb1"
format_stamp: "Formatted at 2026-09-24 11:35:16 on dist-test-slave-cq7c"
server_key: "e990a97b253014782d61c0fc7ed115fe"
server_key_iv: "adb6fb1fa5a313dc4e11bd53de38ec4f"
server_key_version: "encryptionkey@0"
I20260924 11:35:16.648461 17317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "a23347ea14794696874f43624ef61cb1"
format_stamp: "Formatted at 2026-09-24 11:35:16 on dist-test-slave-cq7c"
server_key: "e990a97b253014782d61c0fc7ed115fe"
server_key_iv: "adb6fb1fa5a313dc4e11bd53de38ec4f"
server_key_version: "encryptionkey@0"
I20260924 11:35:16.653854 17317 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 11:35:16.657608 17332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:35:16.659212 17317 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 11:35:16.659480 17317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "a23347ea14794696874f43624ef61cb1"
format_stamp: "Formatted at 2026-09-24 11:35:16 on dist-test-slave-cq7c"
server_key: "e990a97b253014782d61c0fc7ed115fe"
server_key_iv: "adb6fb1fa5a313dc4e11bd53de38ec4f"
server_key_version: "encryptionkey@0"
I20260924 11:35:16.659706 17317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:35:16.676582 17317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:35:16.681049 17317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:35:16.681385 17317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:35:16.700464 17317 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.62:46847
I20260924 11:35:16.700486 17383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.62:46847 every 8 connection(s)
I20260924 11:35:16.701933 17317 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:35:16.705845 17384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:35:16.709988 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 17317
I20260924 11:35:16.710213 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:35:16.710850 6992 external_mini_cluster.cc:1428] Setting key c3ba83510f1a3e52074bead654fb3fd4
I20260924 11:35:16.716897 17384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a23347ea14794696874f43624ef61cb1: Bootstrap starting.
I20260924 11:35:16.720654 17384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a23347ea14794696874f43624ef61cb1: Neither blocks nor log segments found. Creating new log.
I20260924 11:35:16.721936 17384 log.cc:824] T 00000000000000000000000000000000 P a23347ea14794696874f43624ef61cb1: Log is configured to *not* fsync() on all Append() calls
I20260924 11:35:16.724926 17384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a23347ea14794696874f43624ef61cb1: No bootstrap required, opened a new log
Sep 24 11:35:16 dist-test-slave-cq7c krb5kdc[17304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:35:16.736012 17384 raft_consensus.cc:348] T 00000000000000000000000000000000 P a23347ea14794696874f43624ef61cb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a23347ea14794696874f43624ef61cb1" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 46847 } }
I20260924 11:35:16.736398 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:16.713013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49272 (local address 127.6.212.62:46847)
0924 11:35:16.713907 (+ 894us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:16.713920 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:16.713962 (+ 42us) server_negotiation.cc:407] Connection header received
0924 11:35:16.715699 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:16.715721 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:16.716114 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:16.716434 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:16.718223 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:16.719697 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:16.721577 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:16.722361 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:16.722699 (+ 338us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:16.726818 (+ 4119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:35:16.726829 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:35:16.726852 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:35:16.726906 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:35:16.731219 (+ 4313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:35:16.732079 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:35:16.732088 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:35:16.732099 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:35:16.732327 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:35:16.732836 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:35:16.732841 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:35:16.732844 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:35:16.733651 (+ 807us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:35:16.733987 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0924 11:35:16.734415 (+ 428us) server_negotiation.cc:299] Negotiation successful
0924 11:35:16.734930 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":386,"threads_started":1}
I20260924 11:35:16.736509 17384 raft_consensus.cc:374] T 00000000000000000000000000000000 P a23347ea14794696874f43624ef61cb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:35:16.736598 17384 raft_consensus.cc:729] T 00000000000000000000000000000000 P a23347ea14794696874f43624ef61cb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a23347ea14794696874f43624ef61cb1, State: Initialized, Role: FOLLOWER
I20260924 11:35:16.737331 17384 consensus_queue.cc:260] T 00000000000000000000000000000000 P a23347ea14794696874f43624ef61cb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a23347ea14794696874f43624ef61cb1" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 46847 } }
I20260924 11:35:16.737694 17384 raft_consensus.cc:388] T 00000000000000000000000000000000 P a23347ea14794696874f43624ef61cb1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:35:16.737900 17384 raft_consensus.cc:482] T 00000000000000000000000000000000 P a23347ea14794696874f43624ef61cb1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:35:16.738045 17384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a23347ea14794696874f43624ef61cb1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:35:16.740511 17384 raft_consensus.cc:504] T 00000000000000000000000000000000 P a23347ea14794696874f43624ef61cb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a23347ea14794696874f43624ef61cb1" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 46847 } }
I20260924 11:35:16.741029 17384 leader_election.cc:302] T 00000000000000000000000000000000 P a23347ea14794696874f43624ef61cb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a23347ea14794696874f43624ef61cb1; no voters:
I20260924 11:35:16.741717 17384 leader_election.cc:288] T 00000000000000000000000000000000 P a23347ea14794696874f43624ef61cb1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:35:16.741899 17389 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a23347ea14794696874f43624ef61cb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:35:16.742996 17389 raft_consensus.cc:686] T 00000000000000000000000000000000 P a23347ea14794696874f43624ef61cb1 [term 1 LEADER]: Becoming Leader. State: Replica: a23347ea14794696874f43624ef61cb1, State: Running, Role: LEADER
I20260924 11:35:16.743711 17389 consensus_queue.cc:237] T 00000000000000000000000000000000 P a23347ea14794696874f43624ef61cb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a23347ea14794696874f43624ef61cb1" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 46847 } }
I20260924 11:35:16.744057 17384 sys_catalog.cc:564] T 00000000000000000000000000000000 P a23347ea14794696874f43624ef61cb1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:35:16.747187 17390 sys_catalog.cc:455] T 00000000000000000000000000000000 P a23347ea14794696874f43624ef61cb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a23347ea14794696874f43624ef61cb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a23347ea14794696874f43624ef61cb1" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 46847 } } }
I20260924 11:35:16.747148 17391 sys_catalog.cc:455] T 00000000000000000000000000000000 P a23347ea14794696874f43624ef61cb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a23347ea14794696874f43624ef61cb1. Latest consensus state: current_term: 1 leader_uuid: "a23347ea14794696874f43624ef61cb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a23347ea14794696874f43624ef61cb1" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 46847 } } }
I20260924 11:35:16.747658 17390 sys_catalog.cc:458] T 00000000000000000000000000000000 P a23347ea14794696874f43624ef61cb1 [sys.catalog]: This master's current role is: LEADER
I20260924 11:35:16.747691 17391 sys_catalog.cc:458] T 00000000000000000000000000000000 P a23347ea14794696874f43624ef61cb1 [sys.catalog]: This master's current role is: LEADER
I20260924 11:35:16.752871 17398 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:35:16.758381 17398 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:35:16.768468 17398 catalog_manager.cc:1347] Generated new cluster ID: e4ea209bbbae4cc3883da73c1e534042
I20260924 11:35:16.768620 17398 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:35:16.784246 17398 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:35:16.786218 17398 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:35:16.798144 17398 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a23347ea14794696874f43624ef61cb1: Generated new TSK 0
I20260924 11:35:16.799459 17398 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.212.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab.
Entry for principal kudu/127.6.212.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab.
I20260924 11:35:16.868798 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.1:0
--local_ip_for_outbound_sockets=127.6.212.1
--webserver_interface=127.6.212.1
--webserver_port=0
--tserver_master_addrs=127.6.212.62:46847
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab
--principal=kudu/127.6.212.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:35:17.225968 17410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:35:17.226346 17410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:35:17.226522 17410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:35:17.234093 17410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:35:17.234293 17410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:35:17.234474 17410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.1
I20260924 11:35:17.245581 17410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab
--principal=kudu/127.6.212.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.212.1
--webserver_port=0
--tserver_master_addrs=127.6.212.62:46847
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.1
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:35:17.247041 17410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:35:17.248728 17410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:35:17.265074 17415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:35:17.265100 17416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:35:17.265188 17418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:35:17.271929 17410 server_base.cc:1034] running on GCE node
I20260924 11:35:17.272769 17410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:35:17.273903 17410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:35:17.275177 17410 hybrid_clock.cc:648] HybridClock initialized: now 1790249717275133 us; error 77 us; skew 500 ppm
I20260924 11:35:17.275643 17410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:35:17 dist-test-slave-cq7c krb5kdc[17304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249717, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:35:17.279589 17410 init.cc:377] Logged in from keytab as kudu/127.6.212.1@KRBTEST.COM (short username kudu)
I20260924 11:35:17.281414 17410 webserver.cc:460] Webserver started at http://127.6.212.1:45351/ using document root <none> and password file <none>
I20260924 11:35:17.282277 17410 fs_manager.cc:359] Metadata directory not provided
I20260924 11:35:17.282446 17410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:35:17.282791 17410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:35:17.286218 17410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/data/instance:
uuid: "310c06e2ec5c422c905692ea136e36e5"
format_stamp: "Formatted at 2026-09-24 11:35:17 on dist-test-slave-cq7c"
server_key: "a10a385eab6fb2c5bdf47df84ec1bc9f"
server_key_iv: "f57726e7fd4327b46a67de3de0bac2f9"
server_key_version: "encryptionkey@0"
I20260924 11:35:17.287153 17410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance:
uuid: "310c06e2ec5c422c905692ea136e36e5"
format_stamp: "Formatted at 2026-09-24 11:35:17 on dist-test-slave-cq7c"
server_key: "a10a385eab6fb2c5bdf47df84ec1bc9f"
server_key_iv: "f57726e7fd4327b46a67de3de0bac2f9"
server_key_version: "encryptionkey@0"
I20260924 11:35:17.293285 17410 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260924 11:35:17.297616 17425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:35:17.299523 17410 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 11:35:17.299697 17410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/wal
uuid: "310c06e2ec5c422c905692ea136e36e5"
format_stamp: "Formatted at 2026-09-24 11:35:17 on dist-test-slave-cq7c"
server_key: "a10a385eab6fb2c5bdf47df84ec1bc9f"
server_key_iv: "f57726e7fd4327b46a67de3de0bac2f9"
server_key_version: "encryptionkey@0"
I20260924 11:35:17.299901 17410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:35:17.323423 17410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:35:17.328544 17410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:35:17.328822 17410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:35:17.329877 17410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:35:17.331326 17410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:35:17.331485 17410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260924 11:35:17.331589 17410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:35:17.331629 17410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:35:17.368434 17410 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.1:42687
I20260924 11:35:17.368551 17537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.1:42687 every 8 connection(s)
I20260924 11:35:17.369814 17410 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
I20260924 11:35:17.378093 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 17410
I20260924 11:35:17.378355 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance
I20260924 11:35:17.379092 6992 external_mini_cluster.cc:1428] Setting key 8b201274814598ef97de57d264eb96b5
Sep 24 11:35:17 dist-test-slave-cq7c krb5kdc[17304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249717, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:35:17.393530 17540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:17.373072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:41951 (local address 127.6.212.62:46847)
0924 11:35:17.373543 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:17.373548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:17.374773 (+ 1225us) server_negotiation.cc:407] Connection header received
0924 11:35:17.376875 (+ 2102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:17.376881 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:17.376984 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:17.377118 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:17.379022 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:17.379967 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:17.381842 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:17.382322 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:17.382482 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:17.386962 (+ 4480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:35:17.386969 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:35:17.386973 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:35:17.387012 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:35:17.390013 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:35:17.390890 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:35:17.390899 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:35:17.390903 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:35:17.391044 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:35:17.391592 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:35:17.391599 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:35:17.391603 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:35:17.391904 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:35:17.392078 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 11:35:17.392872 (+ 794us) server_negotiation.cc:299] Negotiation successful
0924 11:35:17.393111 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":240,"threads_started":1}
I20260924 11:35:17.394656 17541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:17.373693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:46847 (local address 127.6.212.1:41951)
0924 11:35:17.374436 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:17.374483 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 11:35:17.376551 (+ 2068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:35:17.377406 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:35:17.377415 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:35:17.378019 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:35:17.378811 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:17.378832 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:17.380142 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:17.380156 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:17.381656 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:17.381671 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:17.382612 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:17.382621 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:17.382897 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:17.383740 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:35:17.383771 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:35:17.386605 (+ 2834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:35:17.390200 (+ 3595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:35:17.390211 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:35:17.390233 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:35:17.390746 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:35:17.391175 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:35:17.391180 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:35:17.391184 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:35:17.391471 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:35:17.392050 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:35:17.392059 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:35:17.392460 (+ 401us) client_negotiation.cc:769] Sending connection context
0924 11:35:17.392760 (+ 300us) client_negotiation.cc:240] Negotiation successful
0924 11:35:17.393377 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":280,"threads_started":1}
I20260924 11:35:17.396612 17538 heartbeater.cc:347] Connected to a master server at 127.6.212.62:46847
I20260924 11:35:17.396956 17538 heartbeater.cc:464] Registering TS with master...
I20260924 11:35:17.397780 17538 heartbeater.cc:511] Master 127.6.212.62:46847 requested a full tablet report, sending...
I20260924 11:35:17.400369 17349 ts_manager.cc:194] Registered new tserver with Master: 310c06e2ec5c422c905692ea136e36e5 (127.6.212.1:42687)
I20260924 11:35:17.402872 17349 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.1@KRBTEST.COM'} at 127.6.212.1:41951
WARNING: no policy specified for kudu/127.6.212.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab.
Entry for principal kudu/127.6.212.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab.
I20260924 11:35:17.439577 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.2:0
--local_ip_for_outbound_sockets=127.6.212.2
--webserver_interface=127.6.212.2
--webserver_port=0
--tserver_master_addrs=127.6.212.62:46847
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab
--principal=kudu/127.6.212.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:35:17.790097 17544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:35:17.790400 17544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:35:17.790566 17544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:35:17.798359 17544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:35:17.798472 17544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:35:17.798660 17544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.2
I20260924 11:35:17.808851 17544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab
--principal=kudu/127.6.212.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.212.2
--webserver_port=0
--tserver_master_addrs=127.6.212.62:46847
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.2
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:35:17.810222 17544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:35:17.811700 17544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:35:17.825867 17549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:35:17.825935 17552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:35:17.825908 17550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:35:17.826749 17544 server_base.cc:1034] running on GCE node
I20260924 11:35:17.827458 17544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:35:17.828516 17544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:35:17.829730 17544 hybrid_clock.cc:648] HybridClock initialized: now 1790249717829680 us; error 45 us; skew 500 ppm
I20260924 11:35:17.830071 17544 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:35:17 dist-test-slave-cq7c krb5kdc[17304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249717, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:35:17.833423 17544 init.cc:377] Logged in from keytab as kudu/127.6.212.2@KRBTEST.COM (short username kudu)
I20260924 11:35:17.835002 17544 webserver.cc:460] Webserver started at http://127.6.212.2:37157/ using document root <none> and password file <none>
I20260924 11:35:17.835829 17544 fs_manager.cc:359] Metadata directory not provided
I20260924 11:35:17.835970 17544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:35:17.836304 17544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:35:17.839334 17544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/data/instance:
uuid: "8dc3c94fb56a4ab492dc3ccff54de60c"
format_stamp: "Formatted at 2026-09-24 11:35:17 on dist-test-slave-cq7c"
server_key: "865988b27e6a00a98632be52df17f0a7"
server_key_iv: "13a0d1a2f206bb051b5a94d0c10d9179"
server_key_version: "encryptionkey@0"
I20260924 11:35:17.840204 17544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance:
uuid: "8dc3c94fb56a4ab492dc3ccff54de60c"
format_stamp: "Formatted at 2026-09-24 11:35:17 on dist-test-slave-cq7c"
server_key: "865988b27e6a00a98632be52df17f0a7"
server_key_iv: "13a0d1a2f206bb051b5a94d0c10d9179"
server_key_version: "encryptionkey@0"
I20260924 11:35:17.845731 17544 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 11:35:17.849442 17559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:35:17.850934 17544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 11:35:17.851110 17544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/wal
uuid: "8dc3c94fb56a4ab492dc3ccff54de60c"
format_stamp: "Formatted at 2026-09-24 11:35:17 on dist-test-slave-cq7c"
server_key: "865988b27e6a00a98632be52df17f0a7"
server_key_iv: "13a0d1a2f206bb051b5a94d0c10d9179"
server_key_version: "encryptionkey@0"
I20260924 11:35:17.851337 17544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:35:17.869381 17544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:35:17.873863 17544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:35:17.874177 17544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:35:17.875200 17544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:35:17.876710 17544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:35:17.876816 17544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:35:17.876961 17544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:35:17.877031 17544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:35:17.910485 17544 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.2:41283
I20260924 11:35:17.910533 17671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.2:41283 every 8 connection(s)
I20260924 11:35:17.911883 17544 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
I20260924 11:35:17.916712 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 17544
I20260924 11:35:17.916920 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance
I20260924 11:35:17.917418 6992 external_mini_cluster.cc:1428] Setting key ac73a29854402a83ac189478f53dda8d
Sep 24 11:35:17 dist-test-slave-cq7c krb5kdc[17304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249717, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.2@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:35:17.933885 17674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:17.914836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:49573 (local address 127.6.212.62:46847)
0924 11:35:17.915243 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:17.915247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:17.917386 (+ 2139us) server_negotiation.cc:407] Connection header received
0924 11:35:17.919456 (+ 2070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:17.919461 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:17.919548 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:17.919658 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:17.921211 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:17.921946 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:17.923563 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:17.923981 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:17.924115 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:17.927908 (+ 3793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:35:17.927913 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:35:17.927917 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:35:17.927956 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:35:17.930613 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:35:17.931484 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:35:17.931489 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:35:17.931491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:35:17.931618 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:35:17.932085 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:35:17.932094 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:35:17.932096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:35:17.932374 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:35:17.932530 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 11:35:17.933280 (+ 750us) server_negotiation.cc:299] Negotiation successful
0924 11:35:17.933508 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":199,"threads_started":1}
I20260924 11:35:17.935030 17675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:17.915430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:46847 (local address 127.6.212.2:49573)
0924 11:35:17.917218 (+ 1788us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:17.917258 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 11:35:17.919144 (+ 1886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:35:17.919917 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:35:17.919926 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:35:17.920447 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:35:17.921035 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:17.921053 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:17.922105 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:17.922109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:17.923412 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:17.923433 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:17.924211 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:17.924218 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:17.924470 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:17.925269 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:35:17.925294 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:35:17.927649 (+ 2355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:35:17.930795 (+ 3146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:35:17.930805 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:35:17.930826 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:35:17.931295 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:35:17.931735 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:35:17.931740 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:35:17.931744 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:35:17.931954 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:35:17.932505 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:35:17.932514 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:35:17.932902 (+ 388us) client_negotiation.cc:769] Sending connection context
0924 11:35:17.933184 (+ 282us) client_negotiation.cc:240] Negotiation successful
0924 11:35:17.933775 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":270,"threads_started":1}
I20260924 11:35:17.936637 17672 heartbeater.cc:347] Connected to a master server at 127.6.212.62:46847
I20260924 11:35:17.937000 17672 heartbeater.cc:464] Registering TS with master...
I20260924 11:35:17.937790 17672 heartbeater.cc:511] Master 127.6.212.62:46847 requested a full tablet report, sending...
I20260924 11:35:17.939275 17349 ts_manager.cc:194] Registered new tserver with Master: 8dc3c94fb56a4ab492dc3ccff54de60c (127.6.212.2:41283)
WARNING: no policy specified for kudu/127.6.212.3@KRBTEST.COM; defaulting to no policy
I20260924 11:35:17.940204 17349 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.2@KRBTEST.COM'} at 127.6.212.2:49573
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab.
Entry for principal kudu/127.6.212.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab.
I20260924 11:35:17.975708 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.3:0
--local_ip_for_outbound_sockets=127.6.212.3
--webserver_interface=127.6.212.3
--webserver_port=0
--tserver_master_addrs=127.6.212.62:46847
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab
--principal=kudu/127.6.212.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:39759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:35:18.323839 17678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:35:18.324152 17678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:35:18.324309 17678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:35:18.332566 17678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:35:18.332702 17678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:35:18.332880 17678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.3
I20260924 11:35:18.343168 17678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:39759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab
--principal=kudu/127.6.212.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.212.3
--webserver_port=0
--tserver_master_addrs=127.6.212.62:46847
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.3
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:35:18.344805 17678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:35:18.346282 17678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:35:18.360531 17683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:35:18.360565 17684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:35:18.361163 17686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:35:18.361569 17678 server_base.cc:1034] running on GCE node
I20260924 11:35:18.362253 17678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:35:18.363246 17678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:35:18.364521 17678 hybrid_clock.cc:648] HybridClock initialized: now 1790249718364446 us; error 48 us; skew 500 ppm
I20260924 11:35:18.364924 17678 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:35:18 dist-test-slave-cq7c krb5kdc[17304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249718, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:35:18.368680 17678 init.cc:377] Logged in from keytab as kudu/127.6.212.3@KRBTEST.COM (short username kudu)
I20260924 11:35:18.370703 17678 webserver.cc:460] Webserver started at http://127.6.212.3:37183/ using document root <none> and password file <none>
I20260924 11:35:18.371695 17678 fs_manager.cc:359] Metadata directory not provided
I20260924 11:35:18.371841 17678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:35:18.372225 17678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:35:18.376104 17678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/data/instance:
uuid: "de076e878a7041a0bbbdbd7f56909a4d"
format_stamp: "Formatted at 2026-09-24 11:35:18 on dist-test-slave-cq7c"
server_key: "2b131ed4521f3b579ff1f223d1ee6fe4"
server_key_iv: "8e797f6a45f09716f0432d41a6bfa174"
server_key_version: "encryptionkey@0"
I20260924 11:35:18.376875 17678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance:
uuid: "de076e878a7041a0bbbdbd7f56909a4d"
format_stamp: "Formatted at 2026-09-24 11:35:18 on dist-test-slave-cq7c"
server_key: "2b131ed4521f3b579ff1f223d1ee6fe4"
server_key_iv: "8e797f6a45f09716f0432d41a6bfa174"
server_key_version: "encryptionkey@0"
I20260924 11:35:18.382339 17678 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 11:35:18.386188 17693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:35:18.388046 17678 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:35:18.388260 17678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/wal
uuid: "de076e878a7041a0bbbdbd7f56909a4d"
format_stamp: "Formatted at 2026-09-24 11:35:18 on dist-test-slave-cq7c"
server_key: "2b131ed4521f3b579ff1f223d1ee6fe4"
server_key_iv: "8e797f6a45f09716f0432d41a6bfa174"
server_key_version: "encryptionkey@0"
I20260924 11:35:18.388489 17678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:35:18.406852 17538 heartbeater.cc:503] Master 127.6.212.62:46847 was elected leader, sending a full tablet report...
I20260924 11:35:18.427613 17678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:35:18.432233 17678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:35:18.432544 17678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:35:18.433575 17678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:35:18.435084 17678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:35:18.435189 17678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:35:18.435328 17678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:35:18.435420 17678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260924 11:35:18.471748 17678 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.3:42777
I20260924 11:35:18.471803 17805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.3:42777 every 8 connection(s)
I20260924 11:35:18.473078 17678 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
I20260924 11:35:18.482261 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 17678
I20260924 11:35:18.482537 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance
I20260924 11:35:18.483130 6992 external_mini_cluster.cc:1428] Setting key 013934fe7835117db5dbd809fbc445ce
Sep 24 11:35:18 dist-test-slave-cq7c krb5kdc[17304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249718, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.3@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:35:18.496677 17808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:18.476232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:39517 (local address 127.6.212.62:46847)
0924 11:35:18.476653 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:18.476657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:18.477930 (+ 1273us) server_negotiation.cc:407] Connection header received
0924 11:35:18.480055 (+ 2125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:18.480059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:18.480144 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:18.480244 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:18.482062 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:18.483278 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:18.486293 (+ 3015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:18.486795 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:18.486922 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:18.490661 (+ 3739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:35:18.490667 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:35:18.490671 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:35:18.490706 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:35:18.493498 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:35:18.494364 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:35:18.494368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:35:18.494371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:35:18.494487 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:35:18.494981 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:35:18.494984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:35:18.494986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:35:18.495249 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:35:18.495431 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 11:35:18.496230 (+ 799us) server_negotiation.cc:299] Negotiation successful
0924 11:35:18.496395 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":215,"threads_started":1}
I20260924 11:35:18.497900 17809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:18.476906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:46847 (local address 127.6.212.3:39517)
0924 11:35:18.477736 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:18.477787 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 11:35:18.479747 (+ 1960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:35:18.480488 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:35:18.480497 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:35:18.481145 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:35:18.481785 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:18.481859 (+ 74us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:18.483557 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:18.483565 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:18.486146 (+ 2581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:18.486160 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:18.487025 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:18.487040 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:18.487294 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:18.487998 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:35:18.488023 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:35:18.490355 (+ 2332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:35:18.493696 (+ 3341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:35:18.493704 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:35:18.493718 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:35:18.494250 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:35:18.494677 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:35:18.494680 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:35:18.494683 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:35:18.494877 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:35:18.495436 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:35:18.495450 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:35:18.495873 (+ 423us) client_negotiation.cc:769] Sending connection context
0924 11:35:18.496129 (+ 256us) client_negotiation.cc:240] Negotiation successful
0924 11:35:18.496549 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":343,"threads_started":1}
I20260924 11:35:18.499845 17806 heartbeater.cc:347] Connected to a master server at 127.6.212.62:46847
I20260924 11:35:18.500186 17806 heartbeater.cc:464] Registering TS with master...
I20260924 11:35:18.501045 17806 heartbeater.cc:511] Master 127.6.212.62:46847 requested a full tablet report, sending...
I20260924 11:35:18.502724 17349 ts_manager.cc:194] Registered new tserver with Master: de076e878a7041a0bbbdbd7f56909a4d (127.6.212.3:42777)
I20260924 11:35:18.503639 17349 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.3@KRBTEST.COM'} at 127.6.212.3:39517
I20260924 11:35:18.512993 6992 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:35:18.531867 17808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:18.517866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49282 (local address 127.6.212.62:46847)
0924 11:35:18.518086 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:18.518095 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:18.518586 (+ 491us) server_negotiation.cc:407] Connection header received
0924 11:35:18.518742 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:18.518746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:18.518814 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:18.518898 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:18.520500 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:18.521430 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:18.522781 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:18.523250 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:18.523455 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:18.525822 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:35:18.525827 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:35:18.525831 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:35:18.525867 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:35:18.528790 (+ 2923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:35:18.529702 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:35:18.529706 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:35:18.529709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:35:18.529837 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:35:18.530249 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:35:18.530252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:35:18.530254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:35:18.530615 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:35:18.530807 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0924 11:35:18.531189 (+ 382us) server_negotiation.cc:299] Negotiation successful
0924 11:35:18.531459 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 11:35:18.536792 17349 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49282:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:35:18.544415 17349 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:35:18.576633 17820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:18.558003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.2:41283 (local address 127.0.0.1:42848)
0924 11:35:18.560113 (+ 2110us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:18.560176 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 11:35:18.560418 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:35:18.563958 (+ 3540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:35:18.563969 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:35:18.564013 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:35:18.565313 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:18.565334 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:18.568110 (+ 2776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:18.568119 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:18.571727 (+ 3608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:18.571751 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:18.575018 (+ 3267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:18.575030 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:18.575406 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:18.575498 (+ 92us) client_negotiation.cc:769] Sending connection context
0924 11:35:18.575681 (+ 183us) client_negotiation.cc:240] Negotiation successful
0924 11:35:18.575976 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1745,"mutex_wait_us":1,"thread_start_us":1563,"threads_started":1}
I20260924 11:35:18.578236 17819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:18.558002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42848 (local address 127.6.212.2:41283)
0924 11:35:18.562584 (+ 4582us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:18.562598 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:18.562640 (+ 42us) server_negotiation.cc:407] Connection header received
0924 11:35:18.563044 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:18.563052 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:18.563377 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:18.563586 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:35:18.565677 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:18.567870 (+ 2193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:18.572037 (+ 4167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:18.574307 (+ 2270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:18.574492 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:18.574785 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0924 11:35:18.577487 (+ 2702us) server_negotiation.cc:299] Negotiation successful
0924 11:35:18.577759 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4369,"thread_start_us":4237,"threads_started":1}
I20260924 11:35:18.580246 17821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:18.558264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.3:42777 (local address 127.0.0.1:39388)
0924 11:35:18.563280 (+ 5016us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:18.563329 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 11:35:18.563604 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:35:18.564717 (+ 1113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:35:18.564725 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:35:18.564759 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:35:18.566964 (+ 2205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:18.566984 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:18.569857 (+ 2873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:18.569866 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:18.574842 (+ 4976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:18.574867 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:18.579322 (+ 4455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:18.579328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:18.579567 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:18.579609 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 11:35:18.579702 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 11:35:18.579850 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4921,"thread_start_us":4770,"threads_started":1}
I20260924 11:35:18.580461 17822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:18.558971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39388 (local address 127.6.212.3:42777)
0924 11:35:18.563060 (+ 4089us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:18.563070 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:18.563457 (+ 387us) server_negotiation.cc:407] Connection header received
0924 11:35:18.563862 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:18.563869 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:18.564238 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:18.564483 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:35:18.567243 (+ 2760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:18.569590 (+ 2347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:18.575143 (+ 5553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:18.579102 (+ 3959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:18.579263 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:18.579541 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0924 11:35:18.579868 (+ 327us) server_negotiation.cc:299] Negotiation successful
0924 11:35:18.580069 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3907,"thread_start_us":3739,"threads_started":1}
I20260924 11:35:18.582727 17607 tablet_service.cc:1467] Processing CreateTablet for tablet eb10a5d3033f42ee87ba0160a52060f2 (DEFAULT_TABLE table=test-table [id=a992b257de1343b3b298dcb8001556b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:35:18.584232 17607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb10a5d3033f42ee87ba0160a52060f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:35:18.584546 17741 tablet_service.cc:1467] Processing CreateTablet for tablet eb10a5d3033f42ee87ba0160a52060f2 (DEFAULT_TABLE table=test-table [id=a992b257de1343b3b298dcb8001556b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:35:18.585889 17741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb10a5d3033f42ee87ba0160a52060f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:35:18.589025 17820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:18.563429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:42687 (local address 127.0.0.1:50406)
0924 11:35:18.577052 (+ 13623us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:18.577090 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 11:35:18.577243 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:35:18.580445 (+ 3202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:35:18.580450 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:35:18.580477 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:35:18.581580 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:18.581596 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:18.583903 (+ 2307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:18.583908 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:18.586324 (+ 2416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:18.586341 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:18.588223 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:18.588232 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:18.588416 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:18.588451 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 11:35:18.588554 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 11:35:18.588709 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13470,"thread_start_us":15457,"threads_started":1}
I20260924 11:35:18.589356 17824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:18.563796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50406 (local address 127.6.212.1:42687)
0924 11:35:18.579501 (+ 15705us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:18.579510 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:18.579540 (+ 30us) server_negotiation.cc:407] Connection header received
0924 11:35:18.579646 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:18.579656 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:18.579960 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:18.580187 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:35:18.581779 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:18.583436 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:18.586515 (+ 3079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:18.588034 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:18.588189 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:18.588436 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0924 11:35:18.588619 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 11:35:18.588797 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15490,"thread_start_us":15776,"threads_started":1}
I20260924 11:35:18.595402 17473 tablet_service.cc:1467] Processing CreateTablet for tablet eb10a5d3033f42ee87ba0160a52060f2 (DEFAULT_TABLE table=test-table [id=a992b257de1343b3b298dcb8001556b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:35:18.596956 17473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb10a5d3033f42ee87ba0160a52060f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:35:18.597975 17825 tablet_bootstrap.cc:492] T eb10a5d3033f42ee87ba0160a52060f2 P de076e878a7041a0bbbdbd7f56909a4d: Bootstrap starting.
I20260924 11:35:18.600257 17826 tablet_bootstrap.cc:492] T eb10a5d3033f42ee87ba0160a52060f2 P 8dc3c94fb56a4ab492dc3ccff54de60c: Bootstrap starting.
I20260924 11:35:18.603207 17825 tablet_bootstrap.cc:654] T eb10a5d3033f42ee87ba0160a52060f2 P de076e878a7041a0bbbdbd7f56909a4d: Neither blocks nor log segments found. Creating new log.
I20260924 11:35:18.603928 17826 tablet_bootstrap.cc:654] T eb10a5d3033f42ee87ba0160a52060f2 P 8dc3c94fb56a4ab492dc3ccff54de60c: Neither blocks nor log segments found. Creating new log.
I20260924 11:35:18.604676 17825 log.cc:824] T eb10a5d3033f42ee87ba0160a52060f2 P de076e878a7041a0bbbdbd7f56909a4d: Log is configured to *not* fsync() on all Append() calls
I20260924 11:35:18.605163 17826 log.cc:824] T eb10a5d3033f42ee87ba0160a52060f2 P 8dc3c94fb56a4ab492dc3ccff54de60c: Log is configured to *not* fsync() on all Append() calls
I20260924 11:35:18.607975 17825 tablet_bootstrap.cc:492] T eb10a5d3033f42ee87ba0160a52060f2 P de076e878a7041a0bbbdbd7f56909a4d: No bootstrap required, opened a new log
I20260924 11:35:18.608289 17825 ts_tablet_manager.cc:1397] T eb10a5d3033f42ee87ba0160a52060f2 P de076e878a7041a0bbbdbd7f56909a4d: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260924 11:35:18.608330 17826 tablet_bootstrap.cc:492] T eb10a5d3033f42ee87ba0160a52060f2 P 8dc3c94fb56a4ab492dc3ccff54de60c: No bootstrap required, opened a new log
I20260924 11:35:18.608600 17826 ts_tablet_manager.cc:1397] T eb10a5d3033f42ee87ba0160a52060f2 P 8dc3c94fb56a4ab492dc3ccff54de60c: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260924 11:35:18.611467 17829 tablet_bootstrap.cc:492] T eb10a5d3033f42ee87ba0160a52060f2 P 310c06e2ec5c422c905692ea136e36e5: Bootstrap starting.
I20260924 11:35:18.616366 17829 tablet_bootstrap.cc:654] T eb10a5d3033f42ee87ba0160a52060f2 P 310c06e2ec5c422c905692ea136e36e5: Neither blocks nor log segments found. Creating new log.
I20260924 11:35:18.616700 17825 raft_consensus.cc:348] T eb10a5d3033f42ee87ba0160a52060f2 P de076e878a7041a0bbbdbd7f56909a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dc3c94fb56a4ab492dc3ccff54de60c" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 41283 } } peers { permanent_uuid: "de076e878a7041a0bbbdbd7f56909a4d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 42777 } } peers { permanent_uuid: "310c06e2ec5c422c905692ea136e36e5" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42687 } }
I20260924 11:35:18.617115 17825 raft_consensus.cc:374] T eb10a5d3033f42ee87ba0160a52060f2 P de076e878a7041a0bbbdbd7f56909a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:35:18.617215 17825 raft_consensus.cc:729] T eb10a5d3033f42ee87ba0160a52060f2 P de076e878a7041a0bbbdbd7f56909a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de076e878a7041a0bbbdbd7f56909a4d, State: Initialized, Role: FOLLOWER
I20260924 11:35:18.617923 17829 log.cc:824] T eb10a5d3033f42ee87ba0160a52060f2 P 310c06e2ec5c422c905692ea136e36e5: Log is configured to *not* fsync() on all Append() calls
I20260924 11:35:18.617946 17825 consensus_queue.cc:260] T eb10a5d3033f42ee87ba0160a52060f2 P de076e878a7041a0bbbdbd7f56909a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dc3c94fb56a4ab492dc3ccff54de60c" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 41283 } } peers { permanent_uuid: "de076e878a7041a0bbbdbd7f56909a4d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 42777 } } peers { permanent_uuid: "310c06e2ec5c422c905692ea136e36e5" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42687 } }
I20260924 11:35:18.619131 17806 heartbeater.cc:503] Master 127.6.212.62:46847 was elected leader, sending a full tablet report...
I20260924 11:35:18.620430 17825 ts_tablet_manager.cc:1428] T eb10a5d3033f42ee87ba0160a52060f2 P de076e878a7041a0bbbdbd7f56909a4d: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20260924 11:35:18.620478 17826 raft_consensus.cc:348] T eb10a5d3033f42ee87ba0160a52060f2 P 8dc3c94fb56a4ab492dc3ccff54de60c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dc3c94fb56a4ab492dc3ccff54de60c" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 41283 } } peers { permanent_uuid: "de076e878a7041a0bbbdbd7f56909a4d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 42777 } } peers { permanent_uuid: "310c06e2ec5c422c905692ea136e36e5" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42687 } }
I20260924 11:35:18.620985 17826 raft_consensus.cc:374] T eb10a5d3033f42ee87ba0160a52060f2 P 8dc3c94fb56a4ab492dc3ccff54de60c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:35:18.621109 17826 raft_consensus.cc:729] T eb10a5d3033f42ee87ba0160a52060f2 P 8dc3c94fb56a4ab492dc3ccff54de60c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dc3c94fb56a4ab492dc3ccff54de60c, State: Initialized, Role: FOLLOWER
I20260924 11:35:18.621733 17826 consensus_queue.cc:260] T eb10a5d3033f42ee87ba0160a52060f2 P 8dc3c94fb56a4ab492dc3ccff54de60c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dc3c94fb56a4ab492dc3ccff54de60c" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 41283 } } peers { permanent_uuid: "de076e878a7041a0bbbdbd7f56909a4d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 42777 } } peers { permanent_uuid: "310c06e2ec5c422c905692ea136e36e5" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42687 } }
I20260924 11:35:18.622525 17829 tablet_bootstrap.cc:492] T eb10a5d3033f42ee87ba0160a52060f2 P 310c06e2ec5c422c905692ea136e36e5: No bootstrap required, opened a new log
I20260924 11:35:18.622823 17829 ts_tablet_manager.cc:1397] T eb10a5d3033f42ee87ba0160a52060f2 P 310c06e2ec5c422c905692ea136e36e5: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260924 11:35:18.623526 17826 ts_tablet_manager.cc:1428] T eb10a5d3033f42ee87ba0160a52060f2 P 8dc3c94fb56a4ab492dc3ccff54de60c: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260924 11:35:18.623636 17672 heartbeater.cc:503] Master 127.6.212.62:46847 was elected leader, sending a full tablet report...
I20260924 11:35:18.631444 17829 raft_consensus.cc:348] T eb10a5d3033f42ee87ba0160a52060f2 P 310c06e2ec5c422c905692ea136e36e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dc3c94fb56a4ab492dc3ccff54de60c" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 41283 } } peers { permanent_uuid: "de076e878a7041a0bbbdbd7f56909a4d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 42777 } } peers { permanent_uuid: "310c06e2ec5c422c905692ea136e36e5" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42687 } }
I20260924 11:35:18.631847 17829 raft_consensus.cc:374] T eb10a5d3033f42ee87ba0160a52060f2 P 310c06e2ec5c422c905692ea136e36e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:35:18.631919 17829 raft_consensus.cc:729] T eb10a5d3033f42ee87ba0160a52060f2 P 310c06e2ec5c422c905692ea136e36e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 310c06e2ec5c422c905692ea136e36e5, State: Initialized, Role: FOLLOWER
I20260924 11:35:18.632453 17829 consensus_queue.cc:260] T eb10a5d3033f42ee87ba0160a52060f2 P 310c06e2ec5c422c905692ea136e36e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dc3c94fb56a4ab492dc3ccff54de60c" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 41283 } } peers { permanent_uuid: "de076e878a7041a0bbbdbd7f56909a4d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 42777 } } peers { permanent_uuid: "310c06e2ec5c422c905692ea136e36e5" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42687 } }
I20260924 11:35:18.633839 17829 ts_tablet_manager.cc:1428] T eb10a5d3033f42ee87ba0160a52060f2 P 310c06e2ec5c422c905692ea136e36e5: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
W20260924 11:35:18.664093 17673 tablet.cc:2369] T eb10a5d3033f42ee87ba0160a52060f2 P 8dc3c94fb56a4ab492dc3ccff54de60c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:35:18.724838 17807 tablet.cc:2369] T eb10a5d3033f42ee87ba0160a52060f2 P de076e878a7041a0bbbdbd7f56909a4d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:35:18.774137 17833 raft_consensus.cc:482] T eb10a5d3033f42ee87ba0160a52060f2 P 310c06e2ec5c422c905692ea136e36e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:35:18.774427 17833 raft_consensus.cc:504] T eb10a5d3033f42ee87ba0160a52060f2 P 310c06e2ec5c422c905692ea136e36e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dc3c94fb56a4ab492dc3ccff54de60c" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 41283 } } peers { permanent_uuid: "de076e878a7041a0bbbdbd7f56909a4d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 42777 } } peers { permanent_uuid: "310c06e2ec5c422c905692ea136e36e5" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42687 } }
I20260924 11:35:18.776365 17833 leader_election.cc:288] T eb10a5d3033f42ee87ba0160a52060f2 P 310c06e2ec5c422c905692ea136e36e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8dc3c94fb56a4ab492dc3ccff54de60c (127.6.212.2:41283), de076e878a7041a0bbbdbd7f56909a4d (127.6.212.3:42777)
I20260924 11:35:18.776584 17830 raft_consensus.cc:482] T eb10a5d3033f42ee87ba0160a52060f2 P de076e878a7041a0bbbdbd7f56909a4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:35:18.776899 17830 raft_consensus.cc:504] T eb10a5d3033f42ee87ba0160a52060f2 P de076e878a7041a0bbbdbd7f56909a4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dc3c94fb56a4ab492dc3ccff54de60c" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 41283 } } peers { permanent_uuid: "de076e878a7041a0bbbdbd7f56909a4d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 42777 } } peers { permanent_uuid: "310c06e2ec5c422c905692ea136e36e5" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42687 } }
I20260924 11:35:18.778949 17830 leader_election.cc:288] T eb10a5d3033f42ee87ba0160a52060f2 P de076e878a7041a0bbbdbd7f56909a4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8dc3c94fb56a4ab492dc3ccff54de60c (127.6.212.2:41283), 310c06e2ec5c422c905692ea136e36e5 (127.6.212.1:42687)
I20260924 11:35:18.785568 17819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:18.776931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:47429 (local address 127.6.212.2:41283)
0924 11:35:18.777194 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:18.777203 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:18.777846 (+ 643us) server_negotiation.cc:407] Connection header received
0924 11:35:18.778187 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:18.778193 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:18.778275 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:18.778382 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:35:18.779038 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:18.780251 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:18.782995 (+ 2744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:18.784295 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:18.784395 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:18.784491 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 11:35:18.785149 (+ 658us) server_negotiation.cc:299] Negotiation successful
0924 11:35:18.785301 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260924 11:35:18.786271 17822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:18.777350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:57731 (local address 127.6.212.3:42777)
0924 11:35:18.777538 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:18.777544 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:18.778292 (+ 748us) server_negotiation.cc:407] Connection header received
0924 11:35:18.778496 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:18.778503 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:18.778594 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:18.778654 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:35:18.779452 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:18.780560 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:18.783763 (+ 3203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:18.785121 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:18.785249 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:18.785354 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 11:35:18.785876 (+ 522us) server_negotiation.cc:299] Negotiation successful
0924 11:35:18.786010 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":57}
I20260924 11:35:18.786360 17834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:18.776631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.2:41283 (local address 127.6.212.1:47429)
0924 11:35:18.777707 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:18.777735 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 11:35:18.777985 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:35:18.778406 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:35:18.778411 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:35:18.778447 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:35:18.778856 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:18.778867 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:18.780417 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:18.780422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:18.782813 (+ 2391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:18.782829 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:18.784753 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:18.784760 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:18.784943 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:18.784978 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 11:35:18.785818 (+ 840us) client_negotiation.cc:240] Negotiation successful
0924 11:35:18.785960 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":938,"mutex_wait_us":1,"thread_start_us":764,"threads_started":1}
I20260924 11:35:18.787071 17835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:18.776881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.3:42777 (local address 127.6.212.1:57731)
0924 11:35:18.778198 (+ 1317us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:18.778211 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 11:35:18.778335 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:35:18.778865 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:35:18.778870 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:35:18.778894 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:35:18.779245 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:18.779258 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:18.780731 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:18.780736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:18.783575 (+ 2839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:18.783593 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:18.785491 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:18.785498 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:18.785689 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:18.785730 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 11:35:18.786494 (+ 764us) client_negotiation.cc:240] Negotiation successful
0924 11:35:18.786639 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1159,"mutex_wait_us":600,"thread_start_us":950,"threads_started":1}
I20260924 11:35:18.787691 17627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb10a5d3033f42ee87ba0160a52060f2" candidate_uuid: "310c06e2ec5c422c905692ea136e36e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dc3c94fb56a4ab492dc3ccff54de60c" is_pre_election: true
I20260924 11:35:18.788105 17761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb10a5d3033f42ee87ba0160a52060f2" candidate_uuid: "310c06e2ec5c422c905692ea136e36e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de076e878a7041a0bbbdbd7f56909a4d" is_pre_election: true
I20260924 11:35:18.788370 17627 raft_consensus.cc:2454] T eb10a5d3033f42ee87ba0160a52060f2 P 8dc3c94fb56a4ab492dc3ccff54de60c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 310c06e2ec5c422c905692ea136e36e5 in term 0.
I20260924 11:35:18.788556 17761 raft_consensus.cc:2454] T eb10a5d3033f42ee87ba0160a52060f2 P de076e878a7041a0bbbdbd7f56909a4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 310c06e2ec5c422c905692ea136e36e5 in term 0.
I20260924 11:35:18.789135 17426 leader_election.cc:302] T eb10a5d3033f42ee87ba0160a52060f2 P 310c06e2ec5c422c905692ea136e36e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 310c06e2ec5c422c905692ea136e36e5, 8dc3c94fb56a4ab492dc3ccff54de60c; no voters:
I20260924 11:35:18.789407 17809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:18.779082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.2:41283 (local address 127.6.212.3:43077)
0924 11:35:18.779536 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:18.779554 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:35:18.779671 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:35:18.781952 (+ 2281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:35:18.781957 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:35:18.781985 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:35:18.782205 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:18.782217 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:18.784280 (+ 2063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:18.784285 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:18.786494 (+ 2209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:18.786511 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:18.788669 (+ 2158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:18.788674 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:18.788895 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:18.788931 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 11:35:18.788999 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 11:35:18.789134 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309}
I20260924 11:35:18.789713 17836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:18.779531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:42687 (local address 127.6.212.3:43891)
0924 11:35:18.781909 (+ 2378us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:18.781926 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:35:18.782095 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:35:18.782584 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:35:18.782589 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:35:18.782611 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:35:18.782945 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:18.782955 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:18.784429 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:18.784434 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:18.786715 (+ 2281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:35:18.786728 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:35:18.788547 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:35:18.788555 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:35:18.788765 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:18.788797 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 11:35:18.789184 (+ 387us) client_negotiation.cc:240] Negotiation successful
0924 11:35:18.789341 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2260,"mutex_wait_us":71,"thread_start_us":2072,"threads_started":1}
I20260924 11:35:18.789855 17833 raft_consensus.cc:2781] T eb10a5d3033f42ee87ba0160a52060f2 P 310c06e2ec5c422c905692ea136e36e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:35:18.790020 17824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:18.779972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:43891 (local address 127.6.212.1:42687)
0924 11:35:18.780155 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:18.780162 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:18.782001 (+ 1839us) server_negotiation.cc:407] Connection header received
0924 11:35:18.782257 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:18.782262 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:18.782350 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:18.782413 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:35:18.783114 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:18.784238 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:18.786882 (+ 2644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:18.788161 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:18.788310 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:18.788406 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 11:35:18.788959 (+ 553us) server_negotiation.cc:299] Negotiation successful
0924 11:35:18.789087 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 11:35:18.790035 17833 raft_consensus.cc:482] T eb10a5d3033f42ee87ba0160a52060f2 P 310c06e2ec5c422c905692ea136e36e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:35:18.790246 17833 raft_consensus.cc:3037] T eb10a5d3033f42ee87ba0160a52060f2 P 310c06e2ec5c422c905692ea136e36e5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:35:18.790373 17837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:18.779887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:43077 (local address 127.6.212.2:41283)
0924 11:35:18.781658 (+ 1771us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:18.781664 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:18.781687 (+ 23us) server_negotiation.cc:407] Connection header received
0924 11:35:18.781740 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:18.781745 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:18.781805 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:18.782337 (+ 532us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:35:18.782618 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:18.784074 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:18.787103 (+ 3029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:18.788490 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:18.789716 (+ 1226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:18.789852 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 11:35:18.789901 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 11:35:18.790041 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1614,"thread_start_us":1442,"threads_started":1}
I20260924 11:35:18.790828 17493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb10a5d3033f42ee87ba0160a52060f2" candidate_uuid: "de076e878a7041a0bbbdbd7f56909a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "310c06e2ec5c422c905692ea136e36e5" is_pre_election: true
I20260924 11:35:18.790932 17627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb10a5d3033f42ee87ba0160a52060f2" candidate_uuid: "de076e878a7041a0bbbdbd7f56909a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dc3c94fb56a4ab492dc3ccff54de60c" is_pre_election: true
I20260924 11:35:18.791131 17627 raft_consensus.cc:2454] T eb10a5d3033f42ee87ba0160a52060f2 P 8dc3c94fb56a4ab492dc3ccff54de60c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de076e878a7041a0bbbdbd7f56909a4d in term 0.
I20260924 11:35:18.791582 17694 leader_election.cc:302] T eb10a5d3033f42ee87ba0160a52060f2 P de076e878a7041a0bbbdbd7f56909a4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dc3c94fb56a4ab492dc3ccff54de60c, de076e878a7041a0bbbdbd7f56909a4d; no voters:
I20260924 11:35:18.792084 17830 raft_consensus.cc:2781] T eb10a5d3033f42ee87ba0160a52060f2 P de076e878a7041a0bbbdbd7f56909a4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:35:18.792230 17830 raft_consensus.cc:482] T eb10a5d3033f42ee87ba0160a52060f2 P de076e878a7041a0bbbdbd7f56909a4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:35:18.792349 17830 raft_consensus.cc:3037] T eb10a5d3033f42ee87ba0160a52060f2 P de076e878a7041a0bbbdbd7f56909a4d [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:35:18.793519 17833 raft_consensus.cc:504] T eb10a5d3033f42ee87ba0160a52060f2 P 310c06e2ec5c422c905692ea136e36e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dc3c94fb56a4ab492dc3ccff54de60c" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 41283 } } peers { permanent_uuid: "de076e878a7041a0bbbdbd7f56909a4d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 42777 } } peers { permanent_uuid: "310c06e2ec5c422c905692ea136e36e5" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42687 } }
I20260924 11:35:18.793972 17493 raft_consensus.cc:2379] T eb10a5d3033f42ee87ba0160a52060f2 P 310c06e2ec5c422c905692ea136e36e5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate de076e878a7041a0bbbdbd7f56909a4d in current term 1: Already voted for candidate 310c06e2ec5c422c905692ea136e36e5 in this term.
I20260924 11:35:18.794286 17833 leader_election.cc:288] T eb10a5d3033f42ee87ba0160a52060f2 P 310c06e2ec5c422c905692ea136e36e5 [CANDIDATE]: Term 1 election: Requested vote from peers 8dc3c94fb56a4ab492dc3ccff54de60c (127.6.212.2:41283), de076e878a7041a0bbbdbd7f56909a4d (127.6.212.3:42777)
I20260924 11:35:18.794716 17627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb10a5d3033f42ee87ba0160a52060f2" candidate_uuid: "310c06e2ec5c422c905692ea136e36e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dc3c94fb56a4ab492dc3ccff54de60c"
I20260924 11:35:18.794916 17627 raft_consensus.cc:3037] T eb10a5d3033f42ee87ba0160a52060f2 P 8dc3c94fb56a4ab492dc3ccff54de60c [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:35:18.795411 17830 raft_consensus.cc:504] T eb10a5d3033f42ee87ba0160a52060f2 P de076e878a7041a0bbbdbd7f56909a4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dc3c94fb56a4ab492dc3ccff54de60c" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 41283 } } peers { permanent_uuid: "de076e878a7041a0bbbdbd7f56909a4d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 42777 } } peers { permanent_uuid: "310c06e2ec5c422c905692ea136e36e5" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42687 } }
I20260924 11:35:18.796113 17830 leader_election.cc:288] T eb10a5d3033f42ee87ba0160a52060f2 P de076e878a7041a0bbbdbd7f56909a4d [CANDIDATE]: Term 1 election: Requested vote from peers 8dc3c94fb56a4ab492dc3ccff54de60c (127.6.212.2:41283), 310c06e2ec5c422c905692ea136e36e5 (127.6.212.1:42687)
I20260924 11:35:18.796378 17761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb10a5d3033f42ee87ba0160a52060f2" candidate_uuid: "310c06e2ec5c422c905692ea136e36e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de076e878a7041a0bbbdbd7f56909a4d"
I20260924 11:35:18.796628 17761 raft_consensus.cc:2379] T eb10a5d3033f42ee87ba0160a52060f2 P de076e878a7041a0bbbdbd7f56909a4d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 310c06e2ec5c422c905692ea136e36e5 in current term 1: Already voted for candidate de076e878a7041a0bbbdbd7f56909a4d in this term.
I20260924 11:35:18.796587 17626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb10a5d3033f42ee87ba0160a52060f2" candidate_uuid: "de076e878a7041a0bbbdbd7f56909a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dc3c94fb56a4ab492dc3ccff54de60c"
I20260924 11:35:18.797362 17493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb10a5d3033f42ee87ba0160a52060f2" candidate_uuid: "de076e878a7041a0bbbdbd7f56909a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "310c06e2ec5c422c905692ea136e36e5"
I20260924 11:35:18.797601 17493 raft_consensus.cc:2379] T eb10a5d3033f42ee87ba0160a52060f2 P 310c06e2ec5c422c905692ea136e36e5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate de076e878a7041a0bbbdbd7f56909a4d in current term 1: Already voted for candidate 310c06e2ec5c422c905692ea136e36e5 in this term.
I20260924 11:35:18.797953 17697 leader_election.cc:302] T eb10a5d3033f42ee87ba0160a52060f2 P de076e878a7041a0bbbdbd7f56909a4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: de076e878a7041a0bbbdbd7f56909a4d; no voters: 310c06e2ec5c422c905692ea136e36e5, 8dc3c94fb56a4ab492dc3ccff54de60c
I20260924 11:35:18.798120 17627 raft_consensus.cc:2454] T eb10a5d3033f42ee87ba0160a52060f2 P 8dc3c94fb56a4ab492dc3ccff54de60c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 310c06e2ec5c422c905692ea136e36e5 in term 1.
I20260924 11:35:18.798237 17830 raft_consensus.cc:2726] T eb10a5d3033f42ee87ba0160a52060f2 P de076e878a7041a0bbbdbd7f56909a4d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260924 11:35:18.798550 17426 leader_election.cc:302] T eb10a5d3033f42ee87ba0160a52060f2 P 310c06e2ec5c422c905692ea136e36e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 310c06e2ec5c422c905692ea136e36e5, 8dc3c94fb56a4ab492dc3ccff54de60c; no voters: de076e878a7041a0bbbdbd7f56909a4d
I20260924 11:35:18.798916 17833 raft_consensus.cc:2781] T eb10a5d3033f42ee87ba0160a52060f2 P 310c06e2ec5c422c905692ea136e36e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:35:18.799487 17833 raft_consensus.cc:686] T eb10a5d3033f42ee87ba0160a52060f2 P 310c06e2ec5c422c905692ea136e36e5 [term 1 LEADER]: Becoming Leader. State: Replica: 310c06e2ec5c422c905692ea136e36e5, State: Running, Role: LEADER
I20260924 11:35:18.800213 17833 consensus_queue.cc:237] T eb10a5d3033f42ee87ba0160a52060f2 P 310c06e2ec5c422c905692ea136e36e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dc3c94fb56a4ab492dc3ccff54de60c" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 41283 } } peers { permanent_uuid: "de076e878a7041a0bbbdbd7f56909a4d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 42777 } } peers { permanent_uuid: "310c06e2ec5c422c905692ea136e36e5" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42687 } }
I20260924 11:35:18.804510 17347 catalog_manager.cc:5495] T eb10a5d3033f42ee87ba0160a52060f2 P 310c06e2ec5c422c905692ea136e36e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 310c06e2ec5c422c905692ea136e36e5 (127.6.212.1). New cstate: current_term: 1 leader_uuid: "310c06e2ec5c422c905692ea136e36e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dc3c94fb56a4ab492dc3ccff54de60c" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 41283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de076e878a7041a0bbbdbd7f56909a4d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 42777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "310c06e2ec5c422c905692ea136e36e5" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 42687 } health_report { overall_health: HEALTHY } } }
I20260924 11:35:18.870630 17824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:18.864936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50418 (local address 127.6.212.1:42687)
0924 11:35:18.865174 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:18.865183 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:18.865259 (+ 76us) server_negotiation.cc:407] Connection header received
0924 11:35:18.865442 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:18.865446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:18.865536 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:18.865676 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:35:18.865997 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:18.866782 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:18.868354 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:18.868787 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:18.868928 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:18.869475 (+ 547us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:35:18.870013 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:35:18.870192 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 11:35:18.870245 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 11:35:18.870329 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
W20260924 11:35:18.872452 17539 tablet.cc:2369] T eb10a5d3033f42ee87ba0160a52060f2 P 310c06e2ec5c422c905692ea136e36e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:35:18.879174 17761 raft_consensus.cc:1261] T eb10a5d3033f42ee87ba0160a52060f2 P de076e878a7041a0bbbdbd7f56909a4d [term 1 FOLLOWER]: Refusing update from remote peer 310c06e2ec5c422c905692ea136e36e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:35:18.879206 17627 raft_consensus.cc:1261] T eb10a5d3033f42ee87ba0160a52060f2 P 8dc3c94fb56a4ab492dc3ccff54de60c [term 1 FOLLOWER]: Refusing update from remote peer 310c06e2ec5c422c905692ea136e36e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:35:18.880084 17838 consensus_queue.cc:1035] T eb10a5d3033f42ee87ba0160a52060f2 P 310c06e2ec5c422c905692ea136e36e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dc3c94fb56a4ab492dc3ccff54de60c" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 41283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:35:18.880424 17833 consensus_queue.cc:1035] T eb10a5d3033f42ee87ba0160a52060f2 P 310c06e2ec5c422c905692ea136e36e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de076e878a7041a0bbbdbd7f56909a4d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 42777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:35:18.908483 17347 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:35:18.908793 17347 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:35:18.913046 17347 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a23347ea14794696874f43624ef61cb1: Sending DeleteTablet for 3 replicas of tablet eb10a5d3033f42ee87ba0160a52060f2
I20260924 11:35:18.914379 17607 tablet_service.cc:1514] Processing DeleteTablet for tablet eb10a5d3033f42ee87ba0160a52060f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:35:18 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:42848
I20260924 11:35:18.915666 17741 tablet_service.cc:1514] Processing DeleteTablet for tablet eb10a5d3033f42ee87ba0160a52060f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:35:18 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:39388
I20260924 11:35:18.916570 17852 tablet_replica.cc:330] T eb10a5d3033f42ee87ba0160a52060f2 P de076e878a7041a0bbbdbd7f56909a4d: stopping tablet replica
I20260924 11:35:18.917265 17473 tablet_service.cc:1514] Processing DeleteTablet for tablet eb10a5d3033f42ee87ba0160a52060f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:35:18 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:50406
I20260924 11:35:18.917773 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 17410
I20260924 11:35:18.927022 17852 raft_consensus.cc:2229] T eb10a5d3033f42ee87ba0160a52060f2 P de076e878a7041a0bbbdbd7f56909a4d [term 1 FOLLOWER]: Raft consensus shutting down.
Sep 24 11:35:18 dist-test-slave-cq7c krb5kdc[17304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.1@KRBTEST.COM
I20260924 11:35:18.938614 17852 raft_consensus.cc:2258] T eb10a5d3033f42ee87ba0160a52060f2 P de076e878a7041a0bbbdbd7f56909a4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:35:18.942673 17852 ts_tablet_manager.cc:1905] T eb10a5d3033f42ee87ba0160a52060f2 P de076e878a7041a0bbbdbd7f56909a4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:35:18.943516 17851 tablet_replica.cc:330] T eb10a5d3033f42ee87ba0160a52060f2 P 8dc3c94fb56a4ab492dc3ccff54de60c: stopping tablet replica
I20260924 11:35:18.943965 17851 raft_consensus.cc:2229] T eb10a5d3033f42ee87ba0160a52060f2 P 8dc3c94fb56a4ab492dc3ccff54de60c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:35:18.944504 17851 raft_consensus.cc:2258] T eb10a5d3033f42ee87ba0160a52060f2 P 8dc3c94fb56a4ab492dc3ccff54de60c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:35:18.945281 17853 tablet_replica.cc:330] T eb10a5d3033f42ee87ba0160a52060f2 P 310c06e2ec5c422c905692ea136e36e5: stopping tablet replica
I20260924 11:35:18.945686 17853 raft_consensus.cc:2229] T eb10a5d3033f42ee87ba0160a52060f2 P 310c06e2ec5c422c905692ea136e36e5 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:35:18.946367 17853 raft_consensus.cc:2258] T eb10a5d3033f42ee87ba0160a52060f2 P 310c06e2ec5c422c905692ea136e36e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:35:18.948740 17851 ts_tablet_manager.cc:1905] T eb10a5d3033f42ee87ba0160a52060f2 P 8dc3c94fb56a4ab492dc3ccff54de60c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:35:18.949162 17824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:18.919503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50428 (local address 127.6.212.1:42687)
0924 11:35:18.919742 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:18.919749 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:18.927967 (+ 8218us) server_negotiation.cc:407] Connection header received
0924 11:35:18.928182 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:18.928188 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:18.928273 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:18.928384 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:18.930075 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:18.931224 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:18.933154 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:18.933887 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:18.934012 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:18.940195 (+ 6183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:35:18.940215 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:35:18.940235 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:35:18.940295 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:35:18.944810 (+ 4515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:35:18.945870 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:35:18.945880 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:35:18.945884 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:35:18.946054 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:35:18.946508 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:35:18.946513 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:35:18.946516 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:35:18.947031 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:35:18.947346 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0924 11:35:18.948572 (+ 1226us) server_negotiation.cc:299] Negotiation successful
0924 11:35:18.948809 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260924 11:35:18.950059 17533 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:35:18.952450 17852 ts_tablet_manager.cc:1918] T eb10a5d3033f42ee87ba0160a52060f2 P de076e878a7041a0bbbdbd7f56909a4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:35:18.952785 17852 log.cc:1196] T eb10a5d3033f42ee87ba0160a52060f2 P de076e878a7041a0bbbdbd7f56909a4d: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-2/wal/wals/eb10a5d3033f42ee87ba0160a52060f2
I20260924 11:35:18.953557 17852 ts_tablet_manager.cc:1939] T eb10a5d3033f42ee87ba0160a52060f2 P de076e878a7041a0bbbdbd7f56909a4d: Deleting consensus metadata
I20260924 11:35:18.955415 17851 ts_tablet_manager.cc:1918] T eb10a5d3033f42ee87ba0160a52060f2 P 8dc3c94fb56a4ab492dc3ccff54de60c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:35:18.955590 17851 log.cc:1196] T eb10a5d3033f42ee87ba0160a52060f2 P 8dc3c94fb56a4ab492dc3ccff54de60c: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790249617057452-6992-0/minicluster-data/ts-1/wal/wals/eb10a5d3033f42ee87ba0160a52060f2
I20260924 11:35:18.956190 17851 ts_tablet_manager.cc:1939] T eb10a5d3033f42ee87ba0160a52060f2 P 8dc3c94fb56a4ab492dc3ccff54de60c: Deleting consensus metadata
I20260924 11:35:18.956208 17336 catalog_manager.cc:4841] TS de076e878a7041a0bbbdbd7f56909a4d (127.6.212.3:42777): tablet eb10a5d3033f42ee87ba0160a52060f2 (table test-table [id=a992b257de1343b3b298dcb8001556b7]) successfully deleted
I20260924 11:35:18.958068 17333 catalog_manager.cc:4841] TS 8dc3c94fb56a4ab492dc3ccff54de60c (127.6.212.2:41283): tablet eb10a5d3033f42ee87ba0160a52060f2 (table test-table [id=a992b257de1343b3b298dcb8001556b7]) successfully deleted
I20260924 11:35:19.054181 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 17410
I20260924 11:35:19.073724 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 17544
W20260924 11:35:19.073822 17336 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.212.1:42687 (error 108)
I20260924 11:35:19.075655 17820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:19.075164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:42687 (local address 127.0.0.1:50432)
0924 11:35:19.075340 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:19.075497 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.1:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260924 11:35:19.075941 17336 catalog_manager.cc:4584] TS 310c06e2ec5c422c905692ea136e36e5 (127.6.212.1:42687): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eb10a5d3033f42ee87ba0160a52060f2: Network error: Client connection negotiation failed: client connection to 127.6.212.1:42687: connect: Connection refused (error 111)
Sep 24 11:35:19 dist-test-slave-cq7c krb5kdc[17304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.2@KRBTEST.COM
I20260924 11:35:19.089943 17837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:19.074612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42862 (local address 127.6.212.2:41283)
0924 11:35:19.074900 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:19.074907 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:19.074930 (+ 23us) server_negotiation.cc:407] Connection header received
0924 11:35:19.075094 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:19.075098 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:19.075158 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:19.075282 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:19.076803 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:19.077525 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:19.078792 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:19.079204 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:19.079317 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:19.083788 (+ 4471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:35:19.083804 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:35:19.083822 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:35:19.083858 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:35:19.086905 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:35:19.087845 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:35:19.087853 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:35:19.087855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:35:19.088009 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:35:19.088415 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:35:19.088418 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:35:19.088420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:35:19.088824 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:35:19.089059 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0924 11:35:19.089418 (+ 359us) server_negotiation.cc:299] Negotiation successful
0924 11:35:19.089600 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 11:35:19.090816 17667 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:35:19.129633 17820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:19.129154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:42687 (local address 127.0.0.1:50444)
0924 11:35:19.129389 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:19.129481 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.1:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260924 11:35:19.130775 17820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:19.130351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:42687 (local address 127.0.0.1:50458)
0924 11:35:19.130522 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:19.130652 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.1:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 11:35:19.186961 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 17544
I20260924 11:35:19.191547 17820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:19.190953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:42687 (local address 127.0.0.1:50474)
0924 11:35:19.191202 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:19.191318 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.1:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260924 11:35:19.193013 17820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:19.192540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:42687 (local address 127.0.0.1:50482)
0924 11:35:19.192724 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:19.192818 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.1:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 11:35:19.205224 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 17678
Sep 24 11:35:19 dist-test-slave-cq7c krb5kdc[17304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.3@KRBTEST.COM
I20260924 11:35:19.223136 17822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:19.206161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39396 (local address 127.6.212.3:42777)
0924 11:35:19.206422 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0924 11:35:19.206432 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:35:19.206520 (+ 88us) server_negotiation.cc:407] Connection header received
0924 11:35:19.206758 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:35:19.206765 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:35:19.206872 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:35:19.207016 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:35:19.208505 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:19.209307 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:19.210619 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:35:19.211061 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:35:19.211234 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:35:19.216256 (+ 5022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:35:19.216273 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:35:19.216289 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:35:19.216336 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:35:19.219427 (+ 3091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:35:19.220456 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:35:19.220468 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:35:19.220472 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:35:19.220663 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:35:19.221154 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:35:19.221161 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:35:19.221166 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:35:19.221632 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:35:19.221933 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0924 11:35:19.222257 (+ 324us) server_negotiation.cc:299] Negotiation successful
0924 11:35:19.222477 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260924 11:35:19.223968 17801 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:35:19.273790 17820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:19.273228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:42687 (local address 127.0.0.1:50492)
0924 11:35:19.273480 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:19.273595 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.1:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260924 11:35:19.274992 17820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:19.274492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:42687 (local address 127.0.0.1:50494)
0924 11:35:19.274710 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:19.274810 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.1:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260924 11:35:19.319844 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 17678
I20260924 11:35:19.336822 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 17317
I20260924 11:35:19.338074 17379 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:35:19.449086 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 17317
I20260924 11:35:19.449082 17820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:35:19.448465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:42687 (local address 127.0.0.1:50502)
0924 11:35:19.448668 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 11:35:19.448828 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.1:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
2026-09-24T11:35:19Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4587 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:35:19 dist-test-slave-cq7c krb5kdc[17862](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:35:19 dist-test-slave-cq7c krb5kdc[17862](info): set up 2 sockets
Sep 24 11:35:19 dist-test-slave-cq7c krb5kdc[17862](info): commencing operation
krb5kdc: starting...
W20260924 11:35:20.600296 6992 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.012s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Sep 24 11:35:20 dist-test-slave-cq7c krb5kdc[17862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:35:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:35:20Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.6.212.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.6.212.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.6.212.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/krb5kdc/rangeradmin_127.6.212.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.6.212.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/krb5kdc/rangeradmin_127.6.212.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.6.212.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.6.212.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.6.212.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/krb5kdc/rangerlookup_127.6.212.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.6.212.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/krb5kdc/rangerlookup_127.6.212.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.212.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.212.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.212.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/krb5kdc/HTTP_127.6.212.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.212.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/krb5kdc/HTTP_127.6.212.20@KRBTEST.COM.keytab.
I20260924 11:35:20.979751 6992 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task75Snja/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/postgres -l logfile start
2026-09-24 11:35:23.888 UTC [17892] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 11:35:23.888 UTC [17892] LOG: listening on IPv4 address "127.6.212.20", port 51841
2026-09-24 11:35:23.892 UTC [17892] LOG: listening on Unix socket "/tmp/.s.PGSQL.51841"
2026-09-24 11:35:23.919 UTC [17895] LOG: database system was shut down at 2026-09-24 11:35:22 UTC
2026-09-24 11:35:23.922 UTC [17892] LOG: database system is ready to accept connections
I20260924 11:35:25.004016 6992 mini_postgres.cc:96] Postgres bound to 51841
2026-09-24 11:35:25.024 UTC [17903] FATAL: database "slave" does not exist
127.6.212.20:51841 - accepting connections
I20260924 11:35:25.025321 6992 mini_ranger.cc:161] Starting Ranger...
I20260924 11:35:25.055948 6992 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 11:35:25.629838 6992 mini_ranger.cc:87] Created ranger Postgres database
I20260924 11:35:25.630079 6992 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ranger-admin
2026-09-24 11:35:25,984 [I] DB FLAVOR :POSTGRES
2026-09-24 11:35:25,985 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 11:35:25,985 [I] Checking connection..
2026-09-24 11:35:25,985 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task75Snja/build/asan/bin/postgresql.jar:/tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.212.20:51841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 11:35:26,250 [I] Checking connection passed.
2026-09-24 11:35:26,250 [I] --------- Verifying version history table ---------
2026-09-24 11:35:26,251 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task75Snja/build/asan/bin/postgresql.jar:/tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.212.20:51841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 11:35:26,493 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 11:35:26,493 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task75Snja/build/asan/bin/postgresql.jar:/tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.212.20:51841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 11:35:26,738 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 11:35:26,739 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 11:35:26,739 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task75Snja/build/asan/bin/postgresql.jar:/tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.212.20:51841/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 11:35:27.008 UTC [17967] WARNING: there is no transaction in progress
2026-09-24 11:35:27,015 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 11:35:27,015 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task75Snja/build/asan/bin/postgresql.jar:/tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.212.20:51841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 11:35:27,266 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 11:35:27,266 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 11:35:27,266 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task75Snja/build/asan/bin/postgresql.jar:/tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.212.20:51841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 11:35:27,513 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task75Snja/build/asan/bin/postgresql.jar:/tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.212.20:51841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 11:35:27,770 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task75Snja/build/asan/bin/postgresql.jar:/tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.212.20:51841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-cq7c.c.gcp-upstream.internal','N') ;"
2026-09-24 11:35:28,025 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task75Snja/build/asan/bin/postgresql.jar:/tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.212.20:51841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 11:35:28,258 [I] Table x_portal_user does not exist in database ranger
2026-09-24 11:35:28,258 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task75Snja/build/asan/bin/postgresql.jar:/tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.212.20:51841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 11:35:28,507 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 11:35:28,507 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task75Snja/build/asan/bin/postgresql.jar:/tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.212.20:51841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 11:35:28,742 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task75Snja/build/asan/bin/postgresql.jar:/tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.212.20:51841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 11:35:28,976 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 11:35:28,976 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task75Snja/build/asan/bin/postgresql.jar:/tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.212.20:51841/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 11:35:30.080 UTC [18075] WARNING: there is no transaction in progress
2026-09-24 11:35:30.098 UTC [18075] WARNING: there is no transaction in progress
2026-09-24 11:35:30.117 UTC [18075] WARNING: there is no transaction in progress
2026-09-24 11:35:30.135 UTC [18075] WARNING: there is no transaction in progress
2026-09-24 11:35:30.154 UTC [18075] WARNING: there is no transaction in progress
2026-09-24 11:35:30.175 UTC [18075] WARNING: there is no transaction in progress
2026-09-24 11:35:30.237 UTC [18075] WARNING: there is no transaction in progress
2026-09-24 11:35:30.244 UTC [18075] WARNING: there is no transaction in progress
2026-09-24 11:35:30.261 UTC [18075] WARNING: there is no transaction in progress
2026-09-24 11:35:30.273 UTC [18075] WARNING: there is no transaction in progress
2026-09-24 11:35:30.285 UTC [18075] WARNING: there is no transaction in progress
2026-09-24 11:35:30.298 UTC [18075] WARNING: there is no transaction in progress
2026-09-24 11:35:30.309 UTC [18075] WARNING: there is no transaction in progress
2026-09-24 11:35:30.316 UTC [18075] WARNING: there is no transaction in progress
2026-09-24 11:35:30.323 UTC [18075] WARNING: there is no transaction in progress
2026-09-24 11:35:30.331 UTC [18075] WARNING: there is no transaction in progress
2026-09-24 11:35:31,018 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 11:35:31,019 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task75Snja/build/asan/bin/postgresql.jar:/tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.212.20:51841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 11:35:31,260 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 11:35:31,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task75Snja/build/asan/bin/postgresql.jar:/tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.212.20:51841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 11:35:31,560 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 11:35:31,561 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task75Snja/build/asan/bin/postgresql.jar:/tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.212.20:51841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 11:35:31,826 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 11:35:31,826 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task75Snja/build/asan/bin/postgresql.jar:/tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.212.20:51841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 11:35:32,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task75Snja/build/asan/bin/postgresql.jar:/tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.212.20:51841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 11:35:32,358 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task75Snja/build/asan/bin/postgresql.jar:/tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.212.20:51841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-cq7c.c.gcp-upstream.internal';"
2026-09-24 11:35:32,592 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 11:35:32,593 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task75Snja/build/asan/bin/postgresql.jar:/tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.212.20:51841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 11:35:32,841 [I] DB_PATCHES have already been applied
I20260924 11:35:32.847925 6992 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ranger-admin:/tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task75Snja/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task75Snja/build/asan/bin/postgresql.jar:/tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 11:35:32.848091 6992 mini_ranger.cc:193] Using host: 127.6.212.20
I20260924 11:35:32.865243 6992 mini_ranger.cc:227] Ranger admin URL: http://127.6.212.20:52741
Sep 24, 2026 11:35:33 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 11:35:33 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 11:35:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 11:35:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 11:35:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 11:35:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 11:35:33 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 11:35:34.045599 6992 mini_ranger.cc:160] Time spent starting Ranger: real 9.020s user 0.001s sys 0.048s
Sep 24, 2026 11:35:54 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 11:35:54 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 24, 2026 11:35:54 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 11:35:54 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 11:35:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 11:35:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 11:35:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 11:35:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 11:35:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 11:35:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 11:35:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 11:35:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 11:35:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 11:35:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 11:35:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 11:35:54 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 11:35:54 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 11:35:55 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 11:35:55.598124 6992 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.6.212.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.6.212.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.6.212.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/krb5kdc/rangerkms_127.6.212.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.6.212.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/krb5kdc/rangerkms_127.6.212.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.212.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.6.212.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.212.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/krb5kdc/HTTP_127.6.212.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.212.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/krb5kdc/HTTP_127.6.212.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 11:35:55 dist-test-slave-cq7c krb5kdc[17862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249755, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 11:35:55.801954 6992 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 11:35:55.826820 6992 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 11:35:56.813652 6992 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 11:35:56.813781 6992 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ranger-kms
I20260924 11:35:56.813935 6992 mini_ranger_kms.cc:227] Using postgres at 127.6.212.20:51841
2026-09-24 11:35:57,271 [I] DB FLAVOR :POSTGRES
2026-09-24 11:35:57,271 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 11:35:57,271 [I] Checking connection
2026-09-24 11:35:57,272 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task75Snja/build/asan/bin/postgresql.jar:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.212.20:51841/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 11:35:57,505 [I] connection success
2026-09-24 11:35:57,505 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 11:35:57,505 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 11:35:57,505 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task75Snja/build/asan/bin/postgresql.jar:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.212.20:51841/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 11:35:57,762 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 11:35:57,763 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 11:35:57,763 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 11:35:57,763 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task75Snja/build/asan/bin/postgresql.jar:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.212.20:51841/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 11:35:58,042 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 11:35:58.533913 6992 mini_ranger_kms.cc:310] Created kms service
I20260924 11:35:58.729733 6992 mini_ranger_kms.cc:326] Created kudu user
I20260924 11:35:58.779482 6992 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 11:35:58 dist-test-slave-cq7c krb5kdc[17862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249755, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.6.212.20@KRBTEST.COM
I20260924 11:35:59.226254 6992 mini_ranger_kms.cc:384] Added ranger policy
I20260924 11:35:59.226557 6992 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ranger-kms:/tmp/dist-test-task75Snja/build/asan/bin/postgresql.jar:/tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 11:35:59.226601 6992 mini_ranger_kms.cc:240] Using host: 127.6.212.20
I20260924 11:35:59.246666 6992 mini_ranger_kms.cc:276] Ranger KMS PID: 18255
I20260924 11:35:59.246922 6992 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.6.212.20:47193
Sep 24, 2026 11:36:00 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 11:36:00 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 11:36:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 11:36:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 11:36:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 11:36:00 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 11:36:01.648276 6992 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.846s user 0.119s sys 0.364s
I20260924 11:36:01.648571 6992 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 11:36:01.648785 6992 mini_ranger_kms.cc:401] 127.6.212.20:47193/kms/v1/keys
Sep 24 11:36:01 dist-test-slave-cq7c krb5kdc[17862](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790249761, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.212.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 11:36:02 dist-test-slave-cq7c krb5kdc[17862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249761, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.212.20@KRBTEST.COM for HTTP/127.6.212.20@KRBTEST.COM
Sep 24 11:36:02 dist-test-slave-cq7c krb5kdc[17862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249761, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.212.20@KRBTEST.COM for HTTP/127.6.212.20@KRBTEST.COM
Sep 24 11:36:04 dist-test-slave-cq7c krb5kdc[17862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.6.212.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
I20260924 11:36:04.983220 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:34283
--webserver_interface=127.6.212.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:40489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:34283
--ranger_config_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.212.20:47193/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:36:05.366979 18306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:36:05.367455 18306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:36:05.367614 18306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:36:05.375077 18306 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:36:05.375314 18306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:36:05.375437 18306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:36:05.375555 18306 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:36:05.375655 18306 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:36:05.386339 18306 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:40489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/master-0/wal
--ranger_kms_url=127.6.212.20:47193/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:34283
--ranger_config_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:34283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.212.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:36:05.387820 18306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:36:05.389261 18306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:36:05.402566 18314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:36:05.402701 18311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:36:05.402678 18312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:36:05.403882 18306 server_base.cc:1034] running on GCE node
I20260924 11:36:05.404742 18306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:36:05.406330 18306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:36:05.407541 18306 hybrid_clock.cc:648] HybridClock initialized: now 1790249765407485 us; error 53 us; skew 500 ppm
I20260924 11:36:05.407914 18306 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:36:05 dist-test-slave-cq7c krb5kdc[17862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249765, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:36:05.415577 18306 init.cc:377] Logged in from keytab as kudu/127.6.212.62@KRBTEST.COM (short username kudu)
I20260924 11:36:05.417368 18306 webserver.cc:460] Webserver started at http://127.6.212.62:35099/ using document root <none> and password file <none>
I20260924 11:36:05.418102 18306 fs_manager.cc:359] Metadata directory not provided
I20260924 11:36:05.418246 18306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:36:05.418551 18306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 11:36:05 dist-test-slave-cq7c krb5kdc[17862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249765, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for HTTP/127.6.212.20@KRBTEST.COM
I20260924 11:36:05.477676 18306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "1ed8338601a7447ca6403aec835e80e2"
format_stamp: "Formatted at 2026-09-24 11:36:05 on dist-test-slave-cq7c"
server_key: "18a1992774410d88a03edbfe915c163e"
server_key_iv: "95a4ea08d49830147f6094ff0c03cfa2"
server_key_version: "kuduclusterkey@0"
I20260924 11:36:05.478771 18306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "1ed8338601a7447ca6403aec835e80e2"
format_stamp: "Formatted at 2026-09-24 11:36:05 on dist-test-slave-cq7c"
server_key: "18a1992774410d88a03edbfe915c163e"
server_key_iv: "95a4ea08d49830147f6094ff0c03cfa2"
server_key_version: "kuduclusterkey@0"
I20260924 11:36:05.486508 18306 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260924 11:36:05.507640 18321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:36:05.509702 18306 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 11:36:05.509930 18306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "1ed8338601a7447ca6403aec835e80e2"
format_stamp: "Formatted at 2026-09-24 11:36:05 on dist-test-slave-cq7c"
server_key: "18a1992774410d88a03edbfe915c163e"
server_key_iv: "95a4ea08d49830147f6094ff0c03cfa2"
server_key_version: "kuduclusterkey@0"
I20260924 11:36:05.510196 18306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:36:05.554520 18306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:36:05.559527 18306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:36:05.559911 18306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:36:05.582593 18306 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.62:34283
I20260924 11:36:05.582656 18372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.62:34283 every 8 connection(s)
I20260924 11:36:05.584210 18306 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:36:05.588658 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 18306
I20260924 11:36:05.588886 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:36:05.589591 6992 external_mini_cluster.cc:1428] Setting key 328bb30d5e6b27a28a14f1d4bb763c14
I20260924 11:36:05.589834 18373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:36:05.606153 18373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ed8338601a7447ca6403aec835e80e2: Bootstrap starting.
I20260924 11:36:05.611130 18373 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ed8338601a7447ca6403aec835e80e2: Neither blocks nor log segments found. Creating new log.
I20260924 11:36:05.612571 18373 log.cc:824] T 00000000000000000000000000000000 P 1ed8338601a7447ca6403aec835e80e2: Log is configured to *not* fsync() on all Append() calls
I20260924 11:36:05.616129 18373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ed8338601a7447ca6403aec835e80e2: No bootstrap required, opened a new log
Sep 24 11:36:05 dist-test-slave-cq7c krb5kdc[17862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:36:05.626180 18373 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ed8338601a7447ca6403aec835e80e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed8338601a7447ca6403aec835e80e2" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 34283 } }
I20260924 11:36:05.626717 18373 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ed8338601a7447ca6403aec835e80e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:36:05.626827 18373 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ed8338601a7447ca6403aec835e80e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ed8338601a7447ca6403aec835e80e2, State: Initialized, Role: FOLLOWER
I20260924 11:36:05.627631 18373 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ed8338601a7447ca6403aec835e80e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed8338601a7447ca6403aec835e80e2" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 34283 } }
I20260924 11:36:05.628346 18373 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ed8338601a7447ca6403aec835e80e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:36:05.628509 18373 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ed8338601a7447ca6403aec835e80e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:36:05.628690 18373 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ed8338601a7447ca6403aec835e80e2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:36:05.631585 18373 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ed8338601a7447ca6403aec835e80e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed8338601a7447ca6403aec835e80e2" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 34283 } }
I20260924 11:36:05.632211 18373 leader_election.cc:302] T 00000000000000000000000000000000 P 1ed8338601a7447ca6403aec835e80e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ed8338601a7447ca6403aec835e80e2; no voters:
I20260924 11:36:05.632941 18373 leader_election.cc:288] T 00000000000000000000000000000000 P 1ed8338601a7447ca6403aec835e80e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:36:05.633255 18378 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ed8338601a7447ca6403aec835e80e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:36:05.636521 18376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:36:05.591940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37752 (local address 127.6.212.62:34283)
0924 11:36:05.605924 (+ 13984us) server_negotiation.cc:206] Beginning negotiation
0924 11:36:05.605938 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 11:36:05.608023 (+ 2085us) server_negotiation.cc:407] Connection header received
0924 11:36:05.611234 (+ 3211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:36:05.611269 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:36:05.611916 (+ 647us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:36:05.612502 (+ 586us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:36:05.614199 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:36:05.616211 (+ 2012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:36:05.618210 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:36:05.619333 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:36:05.619572 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:36:05.624537 (+ 4965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:36:05.624548 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:36:05.624573 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:36:05.624625 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:36:05.629219 (+ 4594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:36:05.630171 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:36:05.630180 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:36:05.630190 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:36:05.630396 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:36:05.630829 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:36:05.630834 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:36:05.630838 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:36:05.631657 (+ 819us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:36:05.632006 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0924 11:36:05.633721 (+ 1715us) server_negotiation.cc:299] Negotiation successful
0924 11:36:05.634367 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13720,"thread_start_us":13427,"threads_started":1}
I20260924 11:36:05.640564 18378 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ed8338601a7447ca6403aec835e80e2 [term 1 LEADER]: Becoming Leader. State: Replica: 1ed8338601a7447ca6403aec835e80e2, State: Running, Role: LEADER
I20260924 11:36:05.641310 18378 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ed8338601a7447ca6403aec835e80e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed8338601a7447ca6403aec835e80e2" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 34283 } }
I20260924 11:36:05.648674 18379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ed8338601a7447ca6403aec835e80e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ed8338601a7447ca6403aec835e80e2. Latest consensus state: current_term: 1 leader_uuid: "1ed8338601a7447ca6403aec835e80e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed8338601a7447ca6403aec835e80e2" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 34283 } } }
I20260924 11:36:05.649365 18379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ed8338601a7447ca6403aec835e80e2 [sys.catalog]: This master's current role is: LEADER
I20260924 11:36:05.649163 18373 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ed8338601a7447ca6403aec835e80e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:36:05.650375 18380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ed8338601a7447ca6403aec835e80e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ed8338601a7447ca6403aec835e80e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed8338601a7447ca6403aec835e80e2" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 34283 } } }
I20260924 11:36:05.650758 18380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ed8338601a7447ca6403aec835e80e2 [sys.catalog]: This master's current role is: LEADER
I20260924 11:36:05.660712 18382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:36:05.670068 18382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:36:05.690282 18382 catalog_manager.cc:1347] Generated new cluster ID: 2d0b043fb01944f4a12c67cc574b372c
I20260924 11:36:05.690512 18382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:36:05.708559 18382 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:36:05.710013 18373 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 11:36:05.710654 18382 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:36:05.732267 18382 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ed8338601a7447ca6403aec835e80e2: Generated new TSK 0
I20260924 11:36:05.733695 18382 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.212.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab.
Entry for principal kudu/127.6.212.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab.
I20260924 11:36:06.641842 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.212.20:47193/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.1:0
--local_ip_for_outbound_sockets=127.6.212.1
--webserver_interface=127.6.212.1
--webserver_port=0
--tserver_master_addrs=127.6.212.62:34283
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab
--principal=kudu/127.6.212.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:40489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 11:36:06 dist-test-slave-cq7c krb5kdc[17862](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790249766, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 11:36:07.091162 18418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:36:07.091862 18418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:36:07.092054 18418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:36:07.104844 18418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:36:07.105010 18418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:36:07.105257 18418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.1
I20260924 11:36:07.121912 18418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:40489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.6.212.20:47193/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab
--principal=kudu/127.6.212.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.212.1
--webserver_port=0
--tserver_master_addrs=127.6.212.62:34283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.1
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:36:07.123641 18418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:36:07.125396 18418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:36:07.143584 18428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:36:07.143644 18429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:36:07.147207 18431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:36:07.149006 18418 server_base.cc:1034] running on GCE node
I20260924 11:36:07.149708 18418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:36:07.151018 18418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:36:07.152556 18418 hybrid_clock.cc:648] HybridClock initialized: now 1790249767152476 us; error 66 us; skew 500 ppm
I20260924 11:36:07.152997 18418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:36:07 dist-test-slave-cq7c krb5kdc[17862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249767, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:36:07.157140 18418 init.cc:377] Logged in from keytab as kudu/127.6.212.1@KRBTEST.COM (short username kudu)
I20260924 11:36:07.164342 18418 webserver.cc:460] Webserver started at http://127.6.212.1:40481/ using document root <none> and password file <none>
I20260924 11:36:07.165366 18418 fs_manager.cc:359] Metadata directory not provided
I20260924 11:36:07.165511 18418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:36:07.165896 18418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 11:36:07 dist-test-slave-cq7c krb5kdc[17862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249767, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for HTTP/127.6.212.20@KRBTEST.COM
I20260924 11:36:07.184372 18418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-0/data/instance:
uuid: "ab3377966aa54ca384da6027bfc05efd"
format_stamp: "Formatted at 2026-09-24 11:36:07 on dist-test-slave-cq7c"
server_key: "cd5196e71599ae1288416f8f898e3d04"
server_key_iv: "f795a353f5fbebf2fb4bae453de76add"
server_key_version: "kuduclusterkey@0"
I20260924 11:36:07.185606 18418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance:
uuid: "ab3377966aa54ca384da6027bfc05efd"
format_stamp: "Formatted at 2026-09-24 11:36:07 on dist-test-slave-cq7c"
server_key: "cd5196e71599ae1288416f8f898e3d04"
server_key_iv: "f795a353f5fbebf2fb4bae453de76add"
server_key_version: "kuduclusterkey@0"
I20260924 11:36:07.192507 18418 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20260924 11:36:07.211961 18438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:36:07.226406 18418 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.004s sys 0.001s
I20260924 11:36:07.226737 18418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-0/wal
uuid: "ab3377966aa54ca384da6027bfc05efd"
format_stamp: "Formatted at 2026-09-24 11:36:07 on dist-test-slave-cq7c"
server_key: "cd5196e71599ae1288416f8f898e3d04"
server_key_iv: "f795a353f5fbebf2fb4bae453de76add"
server_key_version: "kuduclusterkey@0"
I20260924 11:36:07.227059 18418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:36:07.289490 18418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:36:07.295589 18418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:36:07.296116 18418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:36:07.301005 18418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:36:07.304158 18418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:36:07.304284 18418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:36:07.304883 18418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:36:07.304962 18418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 11:36:07.407845 18418 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.1:39789
I20260924 11:36:07.409804 18418 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
Sep 24 11:36:07 dist-test-slave-cq7c krb5kdc[17862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249766, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for HTTP/127.6.212.20@KRBTEST.COM
I20260924 11:36:07.407864 18550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.1:39789 every 8 connection(s)
I20260924 11:36:07.418421 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 18418
I20260924 11:36:07.418619 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance
I20260924 11:36:07.419431 6992 external_mini_cluster.cc:1428] Setting key e77bbccd3fb38438a26b45a5a3a4172e
WARNING: no policy specified for kudu/127.6.212.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.2@KRBTEST.COM" created.
Sep 24 11:36:07 dist-test-slave-cq7c krb5kdc[17862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249767, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:36:07.495072 18553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:36:07.419198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:52875 (local address 127.6.212.62:34283)
0924 11:36:07.459819 (+ 40621us) server_negotiation.cc:206] Beginning negotiation
0924 11:36:07.459828 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:36:07.459859 (+ 31us) server_negotiation.cc:407] Connection header received
0924 11:36:07.461728 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:36:07.461737 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:36:07.461872 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:36:07.461983 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:36:07.464456 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:36:07.465809 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:36:07.471547 (+ 5738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:36:07.472558 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:36:07.472678 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:36:07.487090 (+ 14412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:36:07.487098 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:36:07.487103 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:36:07.487151 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:36:07.490900 (+ 3749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:36:07.492079 (+ 1179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:36:07.492085 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:36:07.492089 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:36:07.492261 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:36:07.492851 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:36:07.492857 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:36:07.492860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:36:07.493207 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:36:07.493361 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 11:36:07.494434 (+ 1073us) server_negotiation.cc:299] Negotiation successful
0924 11:36:07.494680 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40425,"thread_start_us":44088,"threads_started":1}
I20260924 11:36:07.497684 18554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:36:07.420020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:34283 (local address 127.6.212.1:52875)
0924 11:36:07.447297 (+ 27277us) negotiation.cc:107] Waiting for socket to connect
0924 11:36:07.447404 (+ 107us) client_negotiation.cc:174] Beginning negotiation
0924 11:36:07.461132 (+ 13728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:36:07.462495 (+ 1363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:36:07.462513 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:36:07.463258 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:36:07.464224 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:36:07.464259 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:36:07.466056 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:36:07.466063 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:36:07.468483 (+ 2420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:36:07.468503 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:36:07.473333 (+ 4830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:36:07.473343 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:36:07.473711 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:36:07.483487 (+ 9776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:36:07.483528 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:36:07.486749 (+ 3221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:36:07.491125 (+ 4376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:36:07.491136 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:36:07.491158 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:36:07.491925 (+ 767us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:36:07.492422 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:36:07.492427 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:36:07.492430 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:36:07.492714 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:36:07.493475 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:36:07.493483 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:36:07.493948 (+ 465us) client_negotiation.cc:769] Sending connection context
0924 11:36:07.495655 (+ 1707us) client_negotiation.cc:240] Negotiation successful
0924 11:36:07.496244 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26822,"thread_start_us":29813,"threads_started":1}
I20260924 11:36:07.501338 18551 heartbeater.cc:347] Connected to a master server at 127.6.212.62:34283
I20260924 11:36:07.501696 18551 heartbeater.cc:464] Registering TS with master...
I20260924 11:36:07.502460 18551 heartbeater.cc:511] Master 127.6.212.62:34283 requested a full tablet report, sending...
I20260924 11:36:07.505125 18338 ts_manager.cc:194] Registered new tserver with Master: ab3377966aa54ca384da6027bfc05efd (127.6.212.1:39789)
I20260924 11:36:07.508013 18338 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.1@KRBTEST.COM'} at 127.6.212.1:52875
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab.
Entry for principal kudu/127.6.212.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab.
W20260924 11:36:07.530005 6992 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.6.212.2: real 0.107s user 0.007s sys 0.013s
I20260924 11:36:07.530972 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.212.20:47193/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.2:0
--local_ip_for_outbound_sockets=127.6.212.2
--webserver_interface=127.6.212.2
--webserver_port=0
--tserver_master_addrs=127.6.212.62:34283
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab
--principal=kudu/127.6.212.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:40489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 11:36:07 dist-test-slave-cq7c krb5kdc[17862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249766, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for HTTP/127.6.212.20@KRBTEST.COM
W20260924 11:36:08.004117 18557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:36:08.004457 18557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:36:08.004642 18557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:36:08.012923 18557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:36:08.013159 18557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:36:08.013398 18557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.2
I20260924 11:36:08.030117 18557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:40489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.6.212.20:47193/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab
--principal=kudu/127.6.212.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.212.2
--webserver_port=0
--tserver_master_addrs=127.6.212.62:34283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.2
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:36:08.032059 18557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:36:08.033993 18557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:36:08.050819 18572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:36:08.051254 18574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:36:08.051424 18571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:36:08.051532 18557 server_base.cc:1034] running on GCE node
I20260924 11:36:08.052621 18557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:36:08.053987 18557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:36:08.055255 18557 hybrid_clock.cc:648] HybridClock initialized: now 1790249768055190 us; error 47 us; skew 500 ppm
I20260924 11:36:08.055672 18557 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:36:08 dist-test-slave-cq7c krb5kdc[17862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249768, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:36:08.059669 18557 init.cc:377] Logged in from keytab as kudu/127.6.212.2@KRBTEST.COM (short username kudu)
I20260924 11:36:08.061427 18557 webserver.cc:460] Webserver started at http://127.6.212.2:35725/ using document root <none> and password file <none>
I20260924 11:36:08.062237 18557 fs_manager.cc:359] Metadata directory not provided
I20260924 11:36:08.062381 18557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:36:08.062892 18557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 11:36:08 dist-test-slave-cq7c krb5kdc[17862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249768, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.2@KRBTEST.COM for HTTP/127.6.212.20@KRBTEST.COM
I20260924 11:36:08.080462 18557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-1/data/instance:
uuid: "c3b8fa25deee4493a5d372dd30e73037"
format_stamp: "Formatted at 2026-09-24 11:36:08 on dist-test-slave-cq7c"
server_key: "7f19791c68e7c3a342b0765222dd88f4"
server_key_iv: "dee4db1dd6ca1fb020da05e22f5b741b"
server_key_version: "kuduclusterkey@0"
I20260924 11:36:08.081683 18557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance:
uuid: "c3b8fa25deee4493a5d372dd30e73037"
format_stamp: "Formatted at 2026-09-24 11:36:08 on dist-test-slave-cq7c"
server_key: "7f19791c68e7c3a342b0765222dd88f4"
server_key_iv: "dee4db1dd6ca1fb020da05e22f5b741b"
server_key_version: "kuduclusterkey@0"
I20260924 11:36:08.089208 18557 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.003s
I20260924 11:36:08.111349 18581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:36:08.113371 18557 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260924 11:36:08.113612 18557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-1/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-1/wal
uuid: "c3b8fa25deee4493a5d372dd30e73037"
format_stamp: "Formatted at 2026-09-24 11:36:08 on dist-test-slave-cq7c"
server_key: "7f19791c68e7c3a342b0765222dd88f4"
server_key_iv: "dee4db1dd6ca1fb020da05e22f5b741b"
server_key_version: "kuduclusterkey@0"
I20260924 11:36:08.113931 18557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:36:08.151916 18557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:36:08.158409 18557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:36:08.158818 18557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:36:08.160148 18557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:36:08.162160 18557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:36:08.162314 18557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:36:08.162468 18557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:36:08.162550 18557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:36:08.219472 18557 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.2:42675
I20260924 11:36:08.219519 18693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.2:42675 every 8 connection(s)
I20260924 11:36:08.221448 18557 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
I20260924 11:36:08.229241 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 18557
I20260924 11:36:08.229422 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance
I20260924 11:36:08.229921 6992 external_mini_cluster.cc:1428] Setting key 5533533642cde989689a5c7808f7a2de
Sep 24 11:36:08 dist-test-slave-cq7c krb5kdc[17862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249768, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.2@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.212.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.3@KRBTEST.COM" created.
I20260924 11:36:08.282862 18696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:36:08.224787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:43701 (local address 127.6.212.62:34283)
0924 11:36:08.253574 (+ 28787us) server_negotiation.cc:206] Beginning negotiation
0924 11:36:08.253582 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:36:08.257512 (+ 3930us) server_negotiation.cc:407] Connection header received
0924 11:36:08.257588 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:36:08.257594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:36:08.257707 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:36:08.257815 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:36:08.261269 (+ 3454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:36:08.262612 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:36:08.265377 (+ 2765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:36:08.266070 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:36:08.266178 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:36:08.274920 (+ 8742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:36:08.274929 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:36:08.274934 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:36:08.274982 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:36:08.278716 (+ 3734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:36:08.279753 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:36:08.279758 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:36:08.279761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:36:08.279902 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:36:08.280458 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:36:08.280463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:36:08.280466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:36:08.280891 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:36:08.281076 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 11:36:08.282184 (+ 1108us) server_negotiation.cc:299] Negotiation successful
0924 11:36:08.282435 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28629,"thread_start_us":28796,"threads_started":1}
I20260924 11:36:08.285468 18697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:36:08.225503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:34283 (local address 127.6.212.2:43701)
0924 11:36:08.254329 (+ 28826us) negotiation.cc:107] Waiting for socket to connect
0924 11:36:08.254402 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0924 11:36:08.257063 (+ 2661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:36:08.259179 (+ 2116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:36:08.259204 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:36:08.260039 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:36:08.260997 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:36:08.261026 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:36:08.262884 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:36:08.262889 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:36:08.265178 (+ 2289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:36:08.265203 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:36:08.266562 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:36:08.266570 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:36:08.266879 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:36:08.271582 (+ 4703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:36:08.271621 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:36:08.274575 (+ 2954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:36:08.278910 (+ 4335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:36:08.278920 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:36:08.278942 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:36:08.279603 (+ 661us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:36:08.280049 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:36:08.280053 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:36:08.280056 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:36:08.280318 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:36:08.281187 (+ 869us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:36:08.281199 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:36:08.281721 (+ 522us) client_negotiation.cc:769] Sending connection context
0924 11:36:08.283452 (+ 1731us) client_negotiation.cc:240] Negotiation successful
0924 11:36:08.284014 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28465,"thread_start_us":28228,"threads_started":1}
I20260924 11:36:08.288028 18694 heartbeater.cc:347] Connected to a master server at 127.6.212.62:34283
I20260924 11:36:08.288379 18694 heartbeater.cc:464] Registering TS with master...
I20260924 11:36:08.289201 18694 heartbeater.cc:511] Master 127.6.212.62:34283 requested a full tablet report, sending...
I20260924 11:36:08.290953 18338 ts_manager.cc:194] Registered new tserver with Master: c3b8fa25deee4493a5d372dd30e73037 (127.6.212.2:42675)
I20260924 11:36:08.291812 18338 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.2@KRBTEST.COM'} at 127.6.212.2:43701
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab.
Entry for principal kudu/127.6.212.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab.
I20260924 11:36:08.315267 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.212.20:47193/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.3:0
--local_ip_for_outbound_sockets=127.6.212.3
--webserver_interface=127.6.212.3
--webserver_port=0
--tserver_master_addrs=127.6.212.62:34283
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab
--principal=kudu/127.6.212.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:40489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:36:08.511924 18551 heartbeater.cc:503] Master 127.6.212.62:34283 was elected leader, sending a full tablet report...
W20260924 11:36:08.730165 18700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:36:08.730475 18700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:36:08.730620 18700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:36:08.739156 18700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:36:08.739266 18700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:36:08.739436 18700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.3
I20260924 11:36:08.749833 18700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:40489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.6.212.20:47193/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab
--principal=kudu/127.6.212.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.212.3
--webserver_port=0
--tserver_master_addrs=127.6.212.62:34283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.3
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:36:08.751161 18700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:36:08.752668 18700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:36:08.767887 18706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:36:08.767637 18705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:36:08.767982 18708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:36:08.769351 18700 server_base.cc:1034] running on GCE node
I20260924 11:36:08.769972 18700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:36:08.770932 18700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:36:08.772202 18700 hybrid_clock.cc:648] HybridClock initialized: now 1790249768772091 us; error 95 us; skew 500 ppm
I20260924 11:36:08.772830 18700 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:36:08 dist-test-slave-cq7c krb5kdc[17862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249768, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:36:08.776217 18700 init.cc:377] Logged in from keytab as kudu/127.6.212.3@KRBTEST.COM (short username kudu)
I20260924 11:36:08.777990 18700 webserver.cc:460] Webserver started at http://127.6.212.3:39481/ using document root <none> and password file <none>
I20260924 11:36:08.778684 18700 fs_manager.cc:359] Metadata directory not provided
I20260924 11:36:08.778785 18700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:36:08.779050 18700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 11:36:08 dist-test-slave-cq7c krb5kdc[17862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249768, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.3@KRBTEST.COM for HTTP/127.6.212.20@KRBTEST.COM
I20260924 11:36:08.793967 18700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-2/data/instance:
uuid: "07725665bd8b4b03a31c284548d5a626"
format_stamp: "Formatted at 2026-09-24 11:36:08 on dist-test-slave-cq7c"
server_key: "822efd1e4414d4e337ab14a137215dd9"
server_key_iv: "0456f20c80ea9c1794d5e97dc6fab702"
server_key_version: "kuduclusterkey@0"
I20260924 11:36:08.794893 18700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance:
uuid: "07725665bd8b4b03a31c284548d5a626"
format_stamp: "Formatted at 2026-09-24 11:36:08 on dist-test-slave-cq7c"
server_key: "822efd1e4414d4e337ab14a137215dd9"
server_key_iv: "0456f20c80ea9c1794d5e97dc6fab702"
server_key_version: "kuduclusterkey@0"
I20260924 11:36:08.801360 18700 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260924 11:36:08.816324 18715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:36:08.818097 18700 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 11:36:08.818342 18700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-2/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-2/wal
uuid: "07725665bd8b4b03a31c284548d5a626"
format_stamp: "Formatted at 2026-09-24 11:36:08 on dist-test-slave-cq7c"
server_key: "822efd1e4414d4e337ab14a137215dd9"
server_key_iv: "0456f20c80ea9c1794d5e97dc6fab702"
server_key_version: "kuduclusterkey@0"
I20260924 11:36:08.818657 18700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:36:08.840967 18700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:36:08.846099 18700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:36:08.846887 18700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:36:08.848112 18700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:36:08.849761 18700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:36:08.849841 18700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:36:08.850019 18700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:36:08.850076 18700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:36:08.889662 18700 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.3:37679
I20260924 11:36:08.889707 18827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.3:37679 every 8 connection(s)
I20260924 11:36:08.890985 18700 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
I20260924 11:36:08.900499 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 18700
I20260924 11:36:08.900658 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance
I20260924 11:36:08.901120 6992 external_mini_cluster.cc:1428] Setting key a804d7346e3efec91d813e8b1d0b77f3
Sep 24 11:36:08 dist-test-slave-cq7c krb5kdc[17862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249768, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.3@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:36:08.910876 18830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:36:08.893539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:58577 (local address 127.6.212.62:34283)
0924 11:36:08.894024 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0924 11:36:08.894028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:36:08.894971 (+ 943us) server_negotiation.cc:407] Connection header received
0924 11:36:08.897368 (+ 2397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:36:08.897373 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:36:08.897483 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:36:08.897601 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:36:08.898966 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:36:08.899767 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:36:08.901561 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:36:08.902077 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:36:08.902197 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:36:08.905716 (+ 3519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:36:08.905721 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:36:08.905724 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:36:08.905764 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:36:08.908219 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:36:08.908971 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:36:08.908976 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:36:08.908978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:36:08.909107 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:36:08.909470 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:36:08.909474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:36:08.909476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:36:08.909732 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:36:08.909851 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 11:36:08.910460 (+ 609us) server_negotiation.cc:299] Negotiation successful
0924 11:36:08.910619 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":256,"threads_started":1}
I20260924 11:36:08.911893 18831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:36:08.894082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:34283 (local address 127.6.212.3:58577)
0924 11:36:08.894805 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0924 11:36:08.894843 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 11:36:08.897094 (+ 2251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:36:08.897835 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:36:08.897843 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:36:08.898262 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:36:08.898801 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:36:08.898817 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:36:08.899933 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:36:08.899938 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:36:08.901406 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:36:08.901418 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:36:08.902309 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:36:08.902327 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:36:08.902633 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:36:08.903414 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:36:08.903452 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:36:08.905524 (+ 2072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:36:08.908366 (+ 2842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:36:08.908372 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:36:08.908384 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:36:08.908794 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:36:08.909205 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:36:08.909208 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:36:08.909211 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:36:08.909377 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:36:08.909851 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:36:08.909857 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:36:08.910137 (+ 280us) client_negotiation.cc:769] Sending connection context
0924 11:36:08.910352 (+ 215us) client_negotiation.cc:240] Negotiation successful
0924 11:36:08.910702 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":317,"threads_started":1}
I20260924 11:36:08.913290 18828 heartbeater.cc:347] Connected to a master server at 127.6.212.62:34283
I20260924 11:36:08.913636 18828 heartbeater.cc:464] Registering TS with master...
I20260924 11:36:08.914325 18828 heartbeater.cc:511] Master 127.6.212.62:34283 requested a full tablet report, sending...
I20260924 11:36:08.915997 18338 ts_manager.cc:194] Registered new tserver with Master: 07725665bd8b4b03a31c284548d5a626 (127.6.212.3:37679)
I20260924 11:36:08.916751 6992 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:36:08.917064 18338 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.3@KRBTEST.COM'} at 127.6.212.3:58577
I20260924 11:36:08.934283 18830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:36:08.922154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35750 (local address 127.6.212.62:34283)
0924 11:36:08.922427 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0924 11:36:08.922431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:36:08.922663 (+ 232us) server_negotiation.cc:407] Connection header received
0924 11:36:08.922820 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:36:08.922825 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:36:08.922909 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:36:08.922994 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:36:08.924424 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:36:08.925157 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:36:08.926452 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:36:08.926846 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:36:08.926940 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:36:08.929286 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:36:08.929290 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:36:08.929294 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:36:08.929322 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:36:08.931782 (+ 2460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:36:08.932456 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:36:08.932460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:36:08.932463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:36:08.932589 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:36:08.932959 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:36:08.932962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:36:08.932964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:36:08.933321 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:36:08.933493 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 11:36:08.933751 (+ 258us) server_negotiation.cc:299] Negotiation successful
0924 11:36:08.933905 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260924 11:36:08.938338 18338 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:36:08.946312 18338 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:36:08.969664 18842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:36:08.959542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60088 (local address 127.6.212.2:42675)
0924 11:36:08.960996 (+ 1454us) server_negotiation.cc:206] Beginning negotiation
0924 11:36:08.961006 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:36:08.961034 (+ 28us) server_negotiation.cc:407] Connection header received
0924 11:36:08.961117 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:36:08.961122 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:36:08.961416 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:36:08.961624 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:36:08.962929 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:36:08.964617 (+ 1688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:36:08.967285 (+ 2668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:36:08.968454 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:36:08.968606 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:36:08.968820 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 11:36:08.969117 (+ 297us) server_negotiation.cc:299] Negotiation successful
0924 11:36:08.969278 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1263,"thread_start_us":1138,"threads_started":1}
I20260924 11:36:08.969776 18841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:36:08.959315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.2:42675 (local address 127.0.0.1:60088)
0924 11:36:08.960582 (+ 1267us) negotiation.cc:107] Waiting for socket to connect
0924 11:36:08.960627 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 11:36:08.960811 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:36:08.961900 (+ 1089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:36:08.961909 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:36:08.961946 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:36:08.962719 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:36:08.962734 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:36:08.964821 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:36:08.964827 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:36:08.967115 (+ 2288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:36:08.967127 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:36:08.968660 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:36:08.968666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:36:08.968896 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:36:08.968932 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 11:36:08.969078 (+ 146us) client_negotiation.cc:240] Negotiation successful
0924 11:36:08.969276 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1025,"mutex_wait_us":3,"thread_start_us":754,"threads_started":1}
I20260924 11:36:08.972790 18843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:36:08.961297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.3:37679 (local address 127.0.0.1:51218)
0924 11:36:08.962209 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0924 11:36:08.962232 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 11:36:08.962444 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:36:08.963663 (+ 1219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:36:08.963668 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:36:08.963693 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:36:08.964128 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:36:08.964141 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:36:08.966312 (+ 2171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:36:08.966317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:36:08.969535 (+ 3218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:36:08.969550 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:36:08.972018 (+ 2468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:36:08.972024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:36:08.972192 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:36:08.972225 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 11:36:08.972320 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 11:36:08.972448 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"mutex_wait_us":28,"thread_start_us":299,"threads_started":1}
I20260924 11:36:08.972949 18844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:36:08.961697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51218 (local address 127.6.212.3:37679)
0924 11:36:08.963002 (+ 1305us) server_negotiation.cc:206] Beginning negotiation
0924 11:36:08.963012 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:36:08.963044 (+ 32us) server_negotiation.cc:407] Connection header received
0924 11:36:08.963131 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:36:08.963138 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:36:08.963400 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:36:08.964318 (+ 918us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:36:08.964567 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:36:08.966188 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:36:08.969823 (+ 3635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:36:08.971796 (+ 1973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:36:08.971954 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:36:08.972192 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0924 11:36:08.972388 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 11:36:08.972566 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1127,"thread_start_us":1507,"threads_started":1}
I20260924 11:36:08.974740 18846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:36:08.963152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:39789 (local address 127.0.0.1:59744)
0924 11:36:08.964423 (+ 1271us) negotiation.cc:107] Waiting for socket to connect
0924 11:36:08.964447 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 11:36:08.964596 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:36:08.965641 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:36:08.965645 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:36:08.965675 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:36:08.966069 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:36:08.966077 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:36:08.969393 (+ 3316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:36:08.969402 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:36:08.971801 (+ 2399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:36:08.971814 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:36:08.973873 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:36:08.973880 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:36:08.974060 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:36:08.974107 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 11:36:08.974230 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 11:36:08.974372 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1071,"thread_start_us":914,"threads_started":1}
I20260924 11:36:08.974803 18845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:36:08.962790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59744 (local address 127.6.212.1:39789)
0924 11:36:08.965183 (+ 2393us) server_negotiation.cc:206] Beginning negotiation
0924 11:36:08.965189 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:36:08.965223 (+ 34us) server_negotiation.cc:407] Connection header received
0924 11:36:08.965297 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:36:08.965301 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:36:08.965461 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:36:08.965656 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:36:08.966437 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:36:08.968810 (+ 2373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:36:08.972010 (+ 3200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:36:08.973667 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:36:08.973823 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:36:08.974062 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0924 11:36:08.974281 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 11:36:08.974447 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2248,"thread_start_us":2070,"threads_started":1}
I20260924 11:36:08.974022 18629 tablet_service.cc:1467] Processing CreateTablet for tablet 38a166dc3e034c82ad1beec543b83d6a (DEFAULT_TABLE table=test-table [id=094945e2deb04792b555bd0bff005e10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:36:08.976985 18629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38a166dc3e034c82ad1beec543b83d6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:36:08.976732 18763 tablet_service.cc:1467] Processing CreateTablet for tablet 38a166dc3e034c82ad1beec543b83d6a (DEFAULT_TABLE table=test-table [id=094945e2deb04792b555bd0bff005e10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:36:08.977972 18763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38a166dc3e034c82ad1beec543b83d6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:36:08.978536 18486 tablet_service.cc:1467] Processing CreateTablet for tablet 38a166dc3e034c82ad1beec543b83d6a (DEFAULT_TABLE table=test-table [id=094945e2deb04792b555bd0bff005e10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:36:08.980063 18486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38a166dc3e034c82ad1beec543b83d6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:36:08.991261 18847 tablet_bootstrap.cc:492] T 38a166dc3e034c82ad1beec543b83d6a P 07725665bd8b4b03a31c284548d5a626: Bootstrap starting.
I20260924 11:36:08.994431 18848 tablet_bootstrap.cc:492] T 38a166dc3e034c82ad1beec543b83d6a P c3b8fa25deee4493a5d372dd30e73037: Bootstrap starting.
I20260924 11:36:08.994534 18847 tablet_bootstrap.cc:654] T 38a166dc3e034c82ad1beec543b83d6a P 07725665bd8b4b03a31c284548d5a626: Neither blocks nor log segments found. Creating new log.
I20260924 11:36:08.995922 18847 log.cc:824] T 38a166dc3e034c82ad1beec543b83d6a P 07725665bd8b4b03a31c284548d5a626: Log is configured to *not* fsync() on all Append() calls
I20260924 11:36:08.998274 18848 tablet_bootstrap.cc:654] T 38a166dc3e034c82ad1beec543b83d6a P c3b8fa25deee4493a5d372dd30e73037: Neither blocks nor log segments found. Creating new log.
I20260924 11:36:08.998512 18849 tablet_bootstrap.cc:492] T 38a166dc3e034c82ad1beec543b83d6a P ab3377966aa54ca384da6027bfc05efd: Bootstrap starting.
I20260924 11:36:08.999760 18847 tablet_bootstrap.cc:492] T 38a166dc3e034c82ad1beec543b83d6a P 07725665bd8b4b03a31c284548d5a626: No bootstrap required, opened a new log
I20260924 11:36:08.999737 18848 log.cc:824] T 38a166dc3e034c82ad1beec543b83d6a P c3b8fa25deee4493a5d372dd30e73037: Log is configured to *not* fsync() on all Append() calls
I20260924 11:36:09.000042 18847 ts_tablet_manager.cc:1397] T 38a166dc3e034c82ad1beec543b83d6a P 07725665bd8b4b03a31c284548d5a626: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260924 11:36:09.002249 18849 tablet_bootstrap.cc:654] T 38a166dc3e034c82ad1beec543b83d6a P ab3377966aa54ca384da6027bfc05efd: Neither blocks nor log segments found. Creating new log.
I20260924 11:36:09.002931 18848 tablet_bootstrap.cc:492] T 38a166dc3e034c82ad1beec543b83d6a P c3b8fa25deee4493a5d372dd30e73037: No bootstrap required, opened a new log
I20260924 11:36:09.003185 18848 ts_tablet_manager.cc:1397] T 38a166dc3e034c82ad1beec543b83d6a P c3b8fa25deee4493a5d372dd30e73037: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260924 11:36:09.003533 18849 log.cc:824] T 38a166dc3e034c82ad1beec543b83d6a P ab3377966aa54ca384da6027bfc05efd: Log is configured to *not* fsync() on all Append() calls
I20260924 11:36:09.006896 18849 tablet_bootstrap.cc:492] T 38a166dc3e034c82ad1beec543b83d6a P ab3377966aa54ca384da6027bfc05efd: No bootstrap required, opened a new log
I20260924 11:36:09.007153 18849 ts_tablet_manager.cc:1397] T 38a166dc3e034c82ad1beec543b83d6a P ab3377966aa54ca384da6027bfc05efd: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260924 11:36:09.009639 18847 raft_consensus.cc:348] T 38a166dc3e034c82ad1beec543b83d6a P 07725665bd8b4b03a31c284548d5a626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b8fa25deee4493a5d372dd30e73037" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 42675 } } peers { permanent_uuid: "07725665bd8b4b03a31c284548d5a626" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 37679 } } peers { permanent_uuid: "ab3377966aa54ca384da6027bfc05efd" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 39789 } }
I20260924 11:36:09.010135 18847 raft_consensus.cc:374] T 38a166dc3e034c82ad1beec543b83d6a P 07725665bd8b4b03a31c284548d5a626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:36:09.010246 18847 raft_consensus.cc:729] T 38a166dc3e034c82ad1beec543b83d6a P 07725665bd8b4b03a31c284548d5a626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07725665bd8b4b03a31c284548d5a626, State: Initialized, Role: FOLLOWER
I20260924 11:36:09.010855 18847 consensus_queue.cc:260] T 38a166dc3e034c82ad1beec543b83d6a P 07725665bd8b4b03a31c284548d5a626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b8fa25deee4493a5d372dd30e73037" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 42675 } } peers { permanent_uuid: "07725665bd8b4b03a31c284548d5a626" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 37679 } } peers { permanent_uuid: "ab3377966aa54ca384da6027bfc05efd" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 39789 } }
I20260924 11:36:09.012557 18828 heartbeater.cc:503] Master 127.6.212.62:34283 was elected leader, sending a full tablet report...
I20260924 11:36:09.013361 18847 ts_tablet_manager.cc:1428] T 38a166dc3e034c82ad1beec543b83d6a P 07725665bd8b4b03a31c284548d5a626: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260924 11:36:09.014122 18848 raft_consensus.cc:348] T 38a166dc3e034c82ad1beec543b83d6a P c3b8fa25deee4493a5d372dd30e73037 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b8fa25deee4493a5d372dd30e73037" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 42675 } } peers { permanent_uuid: "07725665bd8b4b03a31c284548d5a626" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 37679 } } peers { permanent_uuid: "ab3377966aa54ca384da6027bfc05efd" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 39789 } }
I20260924 11:36:09.014547 18848 raft_consensus.cc:374] T 38a166dc3e034c82ad1beec543b83d6a P c3b8fa25deee4493a5d372dd30e73037 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:36:09.014685 18848 raft_consensus.cc:729] T 38a166dc3e034c82ad1beec543b83d6a P c3b8fa25deee4493a5d372dd30e73037 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b8fa25deee4493a5d372dd30e73037, State: Initialized, Role: FOLLOWER
I20260924 11:36:09.015290 18848 consensus_queue.cc:260] T 38a166dc3e034c82ad1beec543b83d6a P c3b8fa25deee4493a5d372dd30e73037 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b8fa25deee4493a5d372dd30e73037" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 42675 } } peers { permanent_uuid: "07725665bd8b4b03a31c284548d5a626" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 37679 } } peers { permanent_uuid: "ab3377966aa54ca384da6027bfc05efd" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 39789 } }
I20260924 11:36:09.018707 18849 raft_consensus.cc:348] T 38a166dc3e034c82ad1beec543b83d6a P ab3377966aa54ca384da6027bfc05efd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b8fa25deee4493a5d372dd30e73037" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 42675 } } peers { permanent_uuid: "07725665bd8b4b03a31c284548d5a626" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 37679 } } peers { permanent_uuid: "ab3377966aa54ca384da6027bfc05efd" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 39789 } }
I20260924 11:36:09.019340 18849 raft_consensus.cc:374] T 38a166dc3e034c82ad1beec543b83d6a P ab3377966aa54ca384da6027bfc05efd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:36:09.019536 18849 raft_consensus.cc:729] T 38a166dc3e034c82ad1beec543b83d6a P ab3377966aa54ca384da6027bfc05efd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab3377966aa54ca384da6027bfc05efd, State: Initialized, Role: FOLLOWER
I20260924 11:36:09.020227 18694 heartbeater.cc:503] Master 127.6.212.62:34283 was elected leader, sending a full tablet report...
I20260924 11:36:09.020224 18849 consensus_queue.cc:260] T 38a166dc3e034c82ad1beec543b83d6a P ab3377966aa54ca384da6027bfc05efd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b8fa25deee4493a5d372dd30e73037" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 42675 } } peers { permanent_uuid: "07725665bd8b4b03a31c284548d5a626" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 37679 } } peers { permanent_uuid: "ab3377966aa54ca384da6027bfc05efd" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 39789 } }
I20260924 11:36:09.021090 18848 ts_tablet_manager.cc:1428] T 38a166dc3e034c82ad1beec543b83d6a P c3b8fa25deee4493a5d372dd30e73037: Time spent starting tablet: real 0.018s user 0.011s sys 0.007s
I20260924 11:36:09.022096 18849 ts_tablet_manager.cc:1428] T 38a166dc3e034c82ad1beec543b83d6a P ab3377966aa54ca384da6027bfc05efd: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260924 11:36:09.084815 18854 raft_consensus.cc:482] T 38a166dc3e034c82ad1beec543b83d6a P c3b8fa25deee4493a5d372dd30e73037 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:36:09.085124 18854 raft_consensus.cc:504] T 38a166dc3e034c82ad1beec543b83d6a P c3b8fa25deee4493a5d372dd30e73037 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b8fa25deee4493a5d372dd30e73037" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 42675 } } peers { permanent_uuid: "07725665bd8b4b03a31c284548d5a626" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 37679 } } peers { permanent_uuid: "ab3377966aa54ca384da6027bfc05efd" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 39789 } }
I20260924 11:36:09.086956 18854 leader_election.cc:288] T 38a166dc3e034c82ad1beec543b83d6a P c3b8fa25deee4493a5d372dd30e73037 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07725665bd8b4b03a31c284548d5a626 (127.6.212.3:37679), ab3377966aa54ca384da6027bfc05efd (127.6.212.1:39789)
I20260924 11:36:09.094869 18844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:36:09.087938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:60735 (local address 127.6.212.3:37679)
0924 11:36:09.088199 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0924 11:36:09.088208 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:36:09.088751 (+ 543us) server_negotiation.cc:407] Connection header received
0924 11:36:09.088982 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:36:09.088991 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:36:09.089092 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:36:09.089198 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:36:09.089897 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:36:09.090854 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:36:09.093020 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:36:09.093938 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:36:09.094102 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:36:09.094210 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 11:36:09.094483 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 11:36:09.094604 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 11:36:09.095052 18856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:36:09.087398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.3:37679 (local address 127.6.212.2:60735)
0924 11:36:09.088654 (+ 1256us) negotiation.cc:107] Waiting for socket to connect
0924 11:36:09.088676 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 11:36:09.088841 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:36:09.089297 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:36:09.089305 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:36:09.089345 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:36:09.089735 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:36:09.089747 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:36:09.091003 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:36:09.091009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:36:09.092863 (+ 1854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:36:09.092875 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:36:09.094160 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:36:09.094166 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:36:09.094322 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:36:09.094346 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 11:36:09.094459 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 11:36:09.094603 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1086,"mutex_wait_us":34,"thread_start_us":775,"threads_started":1}
I20260924 11:36:09.095243 18845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:36:09.088971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:58411 (local address 127.6.212.1:39789)
0924 11:36:09.089204 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 11:36:09.089211 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:36:09.089550 (+ 339us) server_negotiation.cc:407] Connection header received
0924 11:36:09.089688 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:36:09.089696 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:36:09.089772 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:36:09.089858 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:36:09.090425 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:36:09.091434 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:36:09.093271 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:36:09.094156 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:36:09.094408 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:36:09.094532 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 11:36:09.094852 (+ 320us) server_negotiation.cc:299] Negotiation successful
0924 11:36:09.094983 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 11:36:09.095482 18857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:36:09.088658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:39789 (local address 127.6.212.2:58411)
0924 11:36:09.089442 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0924 11:36:09.089456 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 11:36:09.089555 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:36:09.089896 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:36:09.089900 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:36:09.089923 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:36:09.090265 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:36:09.090277 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:36:09.091577 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:36:09.091581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:36:09.093075 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:36:09.093091 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:36:09.094484 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:36:09.094490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:36:09.094667 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:36:09.094690 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 11:36:09.094828 (+ 138us) client_negotiation.cc:240] Negotiation successful
0924 11:36:09.094957 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"mutex_wait_us":19,"thread_start_us":466,"threads_started":1}
I20260924 11:36:09.096544 18783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38a166dc3e034c82ad1beec543b83d6a" candidate_uuid: "c3b8fa25deee4493a5d372dd30e73037" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07725665bd8b4b03a31c284548d5a626" is_pre_election: true
I20260924 11:36:09.096553 18506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38a166dc3e034c82ad1beec543b83d6a" candidate_uuid: "c3b8fa25deee4493a5d372dd30e73037" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab3377966aa54ca384da6027bfc05efd" is_pre_election: true
I20260924 11:36:09.097607 18506 raft_consensus.cc:2454] T 38a166dc3e034c82ad1beec543b83d6a P ab3377966aa54ca384da6027bfc05efd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3b8fa25deee4493a5d372dd30e73037 in term 0.
I20260924 11:36:09.097607 18783 raft_consensus.cc:2454] T 38a166dc3e034c82ad1beec543b83d6a P 07725665bd8b4b03a31c284548d5a626 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3b8fa25deee4493a5d372dd30e73037 in term 0.
I20260924 11:36:09.098330 18582 leader_election.cc:302] T 38a166dc3e034c82ad1beec543b83d6a P c3b8fa25deee4493a5d372dd30e73037 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab3377966aa54ca384da6027bfc05efd, c3b8fa25deee4493a5d372dd30e73037; no voters:
I20260924 11:36:09.098696 18854 raft_consensus.cc:2781] T 38a166dc3e034c82ad1beec543b83d6a P c3b8fa25deee4493a5d372dd30e73037 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:36:09.098846 18854 raft_consensus.cc:482] T 38a166dc3e034c82ad1beec543b83d6a P c3b8fa25deee4493a5d372dd30e73037 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:36:09.098961 18854 raft_consensus.cc:3037] T 38a166dc3e034c82ad1beec543b83d6a P c3b8fa25deee4493a5d372dd30e73037 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:36:09.101488 18854 raft_consensus.cc:504] T 38a166dc3e034c82ad1beec543b83d6a P c3b8fa25deee4493a5d372dd30e73037 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b8fa25deee4493a5d372dd30e73037" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 42675 } } peers { permanent_uuid: "07725665bd8b4b03a31c284548d5a626" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 37679 } } peers { permanent_uuid: "ab3377966aa54ca384da6027bfc05efd" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 39789 } }
I20260924 11:36:09.102124 18854 leader_election.cc:288] T 38a166dc3e034c82ad1beec543b83d6a P c3b8fa25deee4493a5d372dd30e73037 [CANDIDATE]: Term 1 election: Requested vote from peers 07725665bd8b4b03a31c284548d5a626 (127.6.212.3:37679), ab3377966aa54ca384da6027bfc05efd (127.6.212.1:39789)
I20260924 11:36:09.102735 18506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38a166dc3e034c82ad1beec543b83d6a" candidate_uuid: "c3b8fa25deee4493a5d372dd30e73037" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab3377966aa54ca384da6027bfc05efd"
I20260924 11:36:09.102754 18783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38a166dc3e034c82ad1beec543b83d6a" candidate_uuid: "c3b8fa25deee4493a5d372dd30e73037" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07725665bd8b4b03a31c284548d5a626"
I20260924 11:36:09.102963 18506 raft_consensus.cc:3037] T 38a166dc3e034c82ad1beec543b83d6a P ab3377966aa54ca384da6027bfc05efd [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:36:09.102985 18783 raft_consensus.cc:3037] T 38a166dc3e034c82ad1beec543b83d6a P 07725665bd8b4b03a31c284548d5a626 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:36:09.105425 18506 raft_consensus.cc:2454] T 38a166dc3e034c82ad1beec543b83d6a P ab3377966aa54ca384da6027bfc05efd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3b8fa25deee4493a5d372dd30e73037 in term 1.
I20260924 11:36:09.105425 18783 raft_consensus.cc:2454] T 38a166dc3e034c82ad1beec543b83d6a P 07725665bd8b4b03a31c284548d5a626 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3b8fa25deee4493a5d372dd30e73037 in term 1.
I20260924 11:36:09.106034 18582 leader_election.cc:302] T 38a166dc3e034c82ad1beec543b83d6a P c3b8fa25deee4493a5d372dd30e73037 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab3377966aa54ca384da6027bfc05efd, c3b8fa25deee4493a5d372dd30e73037; no voters:
I20260924 11:36:09.106437 18854 raft_consensus.cc:2781] T 38a166dc3e034c82ad1beec543b83d6a P c3b8fa25deee4493a5d372dd30e73037 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:36:09.106917 18854 raft_consensus.cc:686] T 38a166dc3e034c82ad1beec543b83d6a P c3b8fa25deee4493a5d372dd30e73037 [term 1 LEADER]: Becoming Leader. State: Replica: c3b8fa25deee4493a5d372dd30e73037, State: Running, Role: LEADER
I20260924 11:36:09.107496 18854 consensus_queue.cc:237] T 38a166dc3e034c82ad1beec543b83d6a P c3b8fa25deee4493a5d372dd30e73037 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b8fa25deee4493a5d372dd30e73037" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 42675 } } peers { permanent_uuid: "07725665bd8b4b03a31c284548d5a626" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 37679 } } peers { permanent_uuid: "ab3377966aa54ca384da6027bfc05efd" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 39789 } }
I20260924 11:36:09.112579 18338 catalog_manager.cc:5495] T 38a166dc3e034c82ad1beec543b83d6a P c3b8fa25deee4493a5d372dd30e73037 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3b8fa25deee4493a5d372dd30e73037 (127.6.212.2). New cstate: current_term: 1 leader_uuid: "c3b8fa25deee4493a5d372dd30e73037" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b8fa25deee4493a5d372dd30e73037" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 42675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07725665bd8b4b03a31c284548d5a626" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 37679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab3377966aa54ca384da6027bfc05efd" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 39789 } health_report { overall_health: UNKNOWN } } }
W20260924 11:36:09.142693 18829 tablet.cc:2369] T 38a166dc3e034c82ad1beec543b83d6a P 07725665bd8b4b03a31c284548d5a626: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:36:09.165023 18842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:36:09.158810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60102 (local address 127.6.212.2:42675)
0924 11:36:09.159016 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 11:36:09.159021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:36:09.159039 (+ 18us) server_negotiation.cc:407] Connection header received
0924 11:36:09.159301 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:36:09.159305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:36:09.159463 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:36:09.159592 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:36:09.160030 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:36:09.160798 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:36:09.162572 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:36:09.162975 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:36:09.163155 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:36:09.163806 (+ 651us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:36:09.164394 (+ 588us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:36:09.164513 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 11:36:09.164614 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 11:36:09.164683 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 11:36:09.174387 18783 raft_consensus.cc:1261] T 38a166dc3e034c82ad1beec543b83d6a P 07725665bd8b4b03a31c284548d5a626 [term 1 FOLLOWER]: Refusing update from remote peer c3b8fa25deee4493a5d372dd30e73037: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:36:09.174468 18506 raft_consensus.cc:1261] T 38a166dc3e034c82ad1beec543b83d6a P ab3377966aa54ca384da6027bfc05efd [term 1 FOLLOWER]: Refusing update from remote peer c3b8fa25deee4493a5d372dd30e73037: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:36:09.175441 18854 consensus_queue.cc:1035] T 38a166dc3e034c82ad1beec543b83d6a P c3b8fa25deee4493a5d372dd30e73037 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07725665bd8b4b03a31c284548d5a626" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 37679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:36:09.175810 18858 consensus_queue.cc:1035] T 38a166dc3e034c82ad1beec543b83d6a P c3b8fa25deee4493a5d372dd30e73037 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab3377966aa54ca384da6027bfc05efd" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 39789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:36:09.188966 18861 mvcc.cc:205] Tried to move back new op lower bound from 7332863054526824448 to 7332863054268370944. Current Snapshot: MvccSnapshot[applied={T|T < 7332863054526824448}]
I20260924 11:36:09.206238 18337 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:36:09.206517 18337 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:36:09.210228 18337 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1ed8338601a7447ca6403aec835e80e2: Sending DeleteTablet for 3 replicas of tablet 38a166dc3e034c82ad1beec543b83d6a
I20260924 11:36:09.211438 18629 tablet_service.cc:1514] Processing DeleteTablet for tablet 38a166dc3e034c82ad1beec543b83d6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:36:09 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:60088
I20260924 11:36:09.211669 18763 tablet_service.cc:1514] Processing DeleteTablet for tablet 38a166dc3e034c82ad1beec543b83d6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:36:09 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:51218
I20260924 11:36:09.211916 18486 tablet_service.cc:1514] Processing DeleteTablet for tablet 38a166dc3e034c82ad1beec543b83d6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:36:09 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:59744
I20260924 11:36:09.214325 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 18418
I20260924 11:36:09.259837 18873 tablet_replica.cc:330] T 38a166dc3e034c82ad1beec543b83d6a P ab3377966aa54ca384da6027bfc05efd: stopping tablet replica
I20260924 11:36:09.260697 18873 raft_consensus.cc:2229] T 38a166dc3e034c82ad1beec543b83d6a P ab3377966aa54ca384da6027bfc05efd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:36:09.261821 18873 raft_consensus.cc:2258] T 38a166dc3e034c82ad1beec543b83d6a P ab3377966aa54ca384da6027bfc05efd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:36:09.268160 18871 tablet_replica.cc:330] T 38a166dc3e034c82ad1beec543b83d6a P c3b8fa25deee4493a5d372dd30e73037: stopping tablet replica
I20260924 11:36:09.268805 18871 raft_consensus.cc:2229] T 38a166dc3e034c82ad1beec543b83d6a P c3b8fa25deee4493a5d372dd30e73037 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:36:09.269752 18873 ts_tablet_manager.cc:1905] T 38a166dc3e034c82ad1beec543b83d6a P ab3377966aa54ca384da6027bfc05efd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:36:09.269785 18871 raft_consensus.cc:2258] T 38a166dc3e034c82ad1beec543b83d6a P c3b8fa25deee4493a5d372dd30e73037 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:36:09.272068 18871 ts_tablet_manager.cc:1905] T 38a166dc3e034c82ad1beec543b83d6a P c3b8fa25deee4493a5d372dd30e73037: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:36:09.275934 18872 tablet_replica.cc:330] T 38a166dc3e034c82ad1beec543b83d6a P 07725665bd8b4b03a31c284548d5a626: stopping tablet replica
I20260924 11:36:09.276415 18872 raft_consensus.cc:2229] T 38a166dc3e034c82ad1beec543b83d6a P 07725665bd8b4b03a31c284548d5a626 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:36:09.277144 18872 raft_consensus.cc:2258] T 38a166dc3e034c82ad1beec543b83d6a P 07725665bd8b4b03a31c284548d5a626 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:36:09.279472 18873 ts_tablet_manager.cc:1918] T 38a166dc3e034c82ad1beec543b83d6a P ab3377966aa54ca384da6027bfc05efd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:36:09.279561 18872 ts_tablet_manager.cc:1905] T 38a166dc3e034c82ad1beec543b83d6a P 07725665bd8b4b03a31c284548d5a626: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:36:09.279763 18873 log.cc:1196] T 38a166dc3e034c82ad1beec543b83d6a P ab3377966aa54ca384da6027bfc05efd: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-0/wal/wals/38a166dc3e034c82ad1beec543b83d6a
I20260924 11:36:09.280395 18873 ts_tablet_manager.cc:1939] T 38a166dc3e034c82ad1beec543b83d6a P ab3377966aa54ca384da6027bfc05efd: Deleting consensus metadata
I20260924 11:36:09.280807 18871 ts_tablet_manager.cc:1918] T 38a166dc3e034c82ad1beec543b83d6a P c3b8fa25deee4493a5d372dd30e73037: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:36:09.281098 18871 log.cc:1196] T 38a166dc3e034c82ad1beec543b83d6a P c3b8fa25deee4493a5d372dd30e73037: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-1/wal/wals/38a166dc3e034c82ad1beec543b83d6a
I20260924 11:36:09.281728 18871 ts_tablet_manager.cc:1939] T 38a166dc3e034c82ad1beec543b83d6a P c3b8fa25deee4493a5d372dd30e73037: Deleting consensus metadata
I20260924 11:36:09.282943 18322 catalog_manager.cc:4841] TS ab3377966aa54ca384da6027bfc05efd (127.6.212.1:39789): tablet 38a166dc3e034c82ad1beec543b83d6a (table test-table [id=094945e2deb04792b555bd0bff005e10]) successfully deleted
I20260924 11:36:09.283633 18322 catalog_manager.cc:4841] TS c3b8fa25deee4493a5d372dd30e73037 (127.6.212.2:42675): tablet 38a166dc3e034c82ad1beec543b83d6a (table test-table [id=094945e2deb04792b555bd0bff005e10]) successfully deleted
I20260924 11:36:09.286505 18872 ts_tablet_manager.cc:1918] T 38a166dc3e034c82ad1beec543b83d6a P 07725665bd8b4b03a31c284548d5a626: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:36:09.286731 18872 log.cc:1196] T 38a166dc3e034c82ad1beec543b83d6a P 07725665bd8b4b03a31c284548d5a626: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790249617057452-6992-0/minicluster-data/ts-2/wal/wals/38a166dc3e034c82ad1beec543b83d6a
I20260924 11:36:09.287279 18872 ts_tablet_manager.cc:1939] T 38a166dc3e034c82ad1beec543b83d6a P 07725665bd8b4b03a31c284548d5a626: Deleting consensus metadata
I20260924 11:36:09.289077 18322 catalog_manager.cc:4841] TS 07725665bd8b4b03a31c284548d5a626 (127.6.212.3:37679): tablet 38a166dc3e034c82ad1beec543b83d6a (table test-table [id=094945e2deb04792b555bd0bff005e10]) successfully deleted
Sep 24 11:36:09 dist-test-slave-cq7c krb5kdc[17862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.1@KRBTEST.COM
I20260924 11:36:09.298609 18845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:36:09.216161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59756 (local address 127.6.212.1:39789)
0924 11:36:09.216409 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0924 11:36:09.216418 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:36:09.255317 (+ 38899us) server_negotiation.cc:407] Connection header received
0924 11:36:09.255775 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:36:09.255786 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:36:09.255961 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:36:09.256069 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:36:09.262685 (+ 6616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:36:09.264392 (+ 1707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:36:09.283574 (+ 19182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:36:09.284313 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:36:09.284413 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:36:09.291147 (+ 6734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:36:09.291165 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:36:09.291183 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:36:09.291233 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:36:09.295280 (+ 4047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:36:09.296431 (+ 1151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:36:09.296441 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:36:09.296445 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:36:09.296611 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:36:09.297080 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:36:09.297086 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:36:09.297091 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:36:09.297518 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:36:09.297754 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0924 11:36:09.298157 (+ 403us) server_negotiation.cc:299] Negotiation successful
0924 11:36:09.298321 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 11:36:09.299871 18546 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:36:09.398404 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 18418
I20260924 11:36:09.416648 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 18557
Sep 24 11:36:09 dist-test-slave-cq7c krb5kdc[17862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.2@KRBTEST.COM
I20260924 11:36:09.435075 18842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:36:09.417509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60108 (local address 127.6.212.2:42675)
0924 11:36:09.417705 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 11:36:09.417711 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:36:09.417733 (+ 22us) server_negotiation.cc:407] Connection header received
0924 11:36:09.417841 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:36:09.417846 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:36:09.417929 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:36:09.418036 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:36:09.419339 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:36:09.420619 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:36:09.422009 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:36:09.422762 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:36:09.422900 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:36:09.427801 (+ 4901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:36:09.427820 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:36:09.427840 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:36:09.427895 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:36:09.431963 (+ 4068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:36:09.432800 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:36:09.432810 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:36:09.432815 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:36:09.432991 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:36:09.433407 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:36:09.433412 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:36:09.433415 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:36:09.433874 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:36:09.434167 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0924 11:36:09.434458 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 11:36:09.434673 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 11:36:09.435909 18689 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:36:09.550323 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 18557
I20260924 11:36:09.568938 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 18700
Sep 24 11:36:09 dist-test-slave-cq7c krb5kdc[17862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.3@KRBTEST.COM
I20260924 11:36:09.584510 18844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:36:09.569792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51222 (local address 127.6.212.3:37679)
0924 11:36:09.569974 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 11:36:09.569979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:36:09.570014 (+ 35us) server_negotiation.cc:407] Connection header received
0924 11:36:09.570198 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:36:09.570201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:36:09.570288 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:36:09.570373 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:36:09.571725 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:36:09.572483 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:36:09.573704 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:36:09.574083 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:36:09.574176 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:36:09.578680 (+ 4504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:36:09.578697 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:36:09.578713 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:36:09.578756 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:36:09.581500 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:36:09.582244 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:36:09.582266 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:36:09.582269 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:36:09.582439 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:36:09.582789 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:36:09.582797 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:36:09.582801 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:36:09.583312 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:36:09.583615 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0924 11:36:09.583943 (+ 328us) server_negotiation.cc:299] Negotiation successful
0924 11:36:09.584152 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 11:36:09.585211 18823 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:36:09.673069 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 18700
I20260924 11:36:09.689461 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 18306
I20260924 11:36:09.690707 18368 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:36:09.808794 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 18306
I20260924 11:36:09.830878 6992 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 11:36:10.166327 6992 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 11:36:10.166544 6992 mini_ranger.cc:66] Stopping Ranger...
I20260924 11:36:10.512568 6992 mini_ranger.cc:68] Stopped Ranger
2026-09-24 11:36:10.512 UTC [17892] LOG: received smart shutdown request
2026-09-24 11:36:10.516 UTC [17892] LOG: background worker "logical replication launcher" (PID 17901) exited with exit code 1
2026-09-24 11:36:10.517 UTC [17896] LOG: shutting down
2026-09-24 11:36:10.698 UTC [17892] LOG: database system is shut down
2026-09-24T11:36:10Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (51385 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:36:10 dist-test-slave-cq7c krb5kdc[18887](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:36:10 dist-test-slave-cq7c krb5kdc[18887](info): set up 2 sockets
Sep 24 11:36:10 dist-test-slave-cq7c krb5kdc[18887](info): commencing operation
krb5kdc: starting...
W20260924 11:36:11.963905 6992 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Sep 24 11:36:12 dist-test-slave-cq7c krb5kdc[18887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:36:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:36:12Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.6.212.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.6.212.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.6.212.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/rangeradmin_127.6.212.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.6.212.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/rangeradmin_127.6.212.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.6.212.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.6.212.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.6.212.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/rangerlookup_127.6.212.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.6.212.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/rangerlookup_127.6.212.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.212.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.212.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.212.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/HTTP_127.6.212.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.212.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/HTTP_127.6.212.20@KRBTEST.COM.keytab.
I20260924 11:36:12.339766 6992 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task75Snja/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/postgres -l logfile start
2026-09-24 11:36:15.221 UTC [18917] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 11:36:15.221 UTC [18917] LOG: listening on IPv4 address "127.6.212.20", port 35553
2026-09-24 11:36:15.225 UTC [18917] LOG: listening on Unix socket "/tmp/.s.PGSQL.35553"
2026-09-24 11:36:15.248 UTC [18920] LOG: database system was shut down at 2026-09-24 11:36:13 UTC
2026-09-24 11:36:15.252 UTC [18917] LOG: database system is ready to accept connections
I20260924 11:36:16.248005 6992 mini_postgres.cc:96] Postgres bound to 35553
2026-09-24 11:36:16.267 UTC [18928] FATAL: database "slave" does not exist
127.6.212.20:35553 - accepting connections
I20260924 11:36:16.268474 6992 mini_ranger.cc:161] Starting Ranger...
I20260924 11:36:16.296663 6992 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 11:36:16.868958 6992 mini_ranger.cc:87] Created ranger Postgres database
I20260924 11:36:16.869102 6992 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ranger-admin
2026-09-24 11:36:17,190 [I] DB FLAVOR :POSTGRES
2026-09-24 11:36:17,190 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 11:36:17,190 [I] Checking connection..
2026-09-24 11:36:17,191 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task75Snja/build/asan/bin/postgresql.jar:/tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.212.20:35553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 11:36:17,455 [I] Checking connection passed.
2026-09-24 11:36:17,455 [I] --------- Verifying version history table ---------
2026-09-24 11:36:17,455 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task75Snja/build/asan/bin/postgresql.jar:/tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.212.20:35553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 11:36:17,697 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 11:36:17,698 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task75Snja/build/asan/bin/postgresql.jar:/tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.212.20:35553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 11:36:17,968 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 11:36:17,968 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 11:36:17,968 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task75Snja/build/asan/bin/postgresql.jar:/tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.212.20:35553/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 11:36:18.207 UTC [18992] WARNING: there is no transaction in progress
2026-09-24 11:36:18,214 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 11:36:18,214 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task75Snja/build/asan/bin/postgresql.jar:/tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.212.20:35553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 11:36:18,483 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 11:36:18,483 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 11:36:18,484 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task75Snja/build/asan/bin/postgresql.jar:/tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.212.20:35553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 11:36:18,754 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task75Snja/build/asan/bin/postgresql.jar:/tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.212.20:35553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 11:36:18,992 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task75Snja/build/asan/bin/postgresql.jar:/tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.212.20:35553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-cq7c.c.gcp-upstream.internal','N') ;"
2026-09-24 11:36:19,226 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task75Snja/build/asan/bin/postgresql.jar:/tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.212.20:35553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 11:36:19,493 [I] Table x_portal_user does not exist in database ranger
2026-09-24 11:36:19,493 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task75Snja/build/asan/bin/postgresql.jar:/tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.212.20:35553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 11:36:19,746 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 11:36:19,747 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task75Snja/build/asan/bin/postgresql.jar:/tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.212.20:35553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 11:36:20,018 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task75Snja/build/asan/bin/postgresql.jar:/tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.212.20:35553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 11:36:20,273 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 11:36:20,274 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task75Snja/build/asan/bin/postgresql.jar:/tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.212.20:35553/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 11:36:21.391 UTC [19100] WARNING: there is no transaction in progress
2026-09-24 11:36:21.409 UTC [19100] WARNING: there is no transaction in progress
2026-09-24 11:36:21.428 UTC [19100] WARNING: there is no transaction in progress
2026-09-24 11:36:21.445 UTC [19100] WARNING: there is no transaction in progress
2026-09-24 11:36:21.465 UTC [19100] WARNING: there is no transaction in progress
2026-09-24 11:36:21.491 UTC [19100] WARNING: there is no transaction in progress
2026-09-24 11:36:21.563 UTC [19100] WARNING: there is no transaction in progress
2026-09-24 11:36:21.571 UTC [19100] WARNING: there is no transaction in progress
2026-09-24 11:36:21.588 UTC [19100] WARNING: there is no transaction in progress
2026-09-24 11:36:21.599 UTC [19100] WARNING: there is no transaction in progress
2026-09-24 11:36:21.610 UTC [19100] WARNING: there is no transaction in progress
2026-09-24 11:36:21.625 UTC [19100] WARNING: there is no transaction in progress
2026-09-24 11:36:21.635 UTC [19100] WARNING: there is no transaction in progress
2026-09-24 11:36:21.643 UTC [19100] WARNING: there is no transaction in progress
2026-09-24 11:36:21.651 UTC [19100] WARNING: there is no transaction in progress
2026-09-24 11:36:21.658 UTC [19100] WARNING: there is no transaction in progress
2026-09-24 11:36:22,433 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 11:36:22,434 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task75Snja/build/asan/bin/postgresql.jar:/tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.212.20:35553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 11:36:22,698 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 11:36:22,698 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task75Snja/build/asan/bin/postgresql.jar:/tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.212.20:35553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 11:36:22,993 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 11:36:22,994 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task75Snja/build/asan/bin/postgresql.jar:/tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.212.20:35553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 11:36:23,280 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 11:36:23,281 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task75Snja/build/asan/bin/postgresql.jar:/tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.212.20:35553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 11:36:23,552 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task75Snja/build/asan/bin/postgresql.jar:/tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.212.20:35553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 11:36:23,803 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task75Snja/build/asan/bin/postgresql.jar:/tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.212.20:35553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-cq7c.c.gcp-upstream.internal';"
2026-09-24 11:36:24,063 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 11:36:24,063 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task75Snja/build/asan/bin/postgresql.jar:/tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.212.20:35553/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 11:36:24,297 [I] DB_PATCHES have already been applied
I20260924 11:36:24.304630 6992 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ranger-admin:/tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task75Snja/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task75Snja/build/asan/bin/postgresql.jar:/tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 11:36:24.304735 6992 mini_ranger.cc:193] Using host: 127.6.212.20
I20260924 11:36:24.320542 6992 mini_ranger.cc:227] Ranger admin URL: http://127.6.212.20:44531
Sep 24, 2026 11:36:24 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 11:36:24 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 11:36:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 11:36:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 11:36:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 11:36:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 11:36:25 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 11:36:25.489362 6992 mini_ranger.cc:160] Time spent starting Ranger: real 9.221s user 0.003s sys 0.042s
Sep 24, 2026 11:36:46 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 11:36:46 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 24, 2026 11:36:46 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 11:36:46 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 11:36:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 11:36:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 11:36:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 11:36:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 11:36:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 11:36:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 11:36:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 11:36:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 11:36:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 11:36:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 11:36:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 11:36:46 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 11:36:46 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 11:36:47 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 11:36:48.409071 6992 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.6.212.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.6.212.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.6.212.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/rangerkms_127.6.212.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.6.212.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/rangerkms_127.6.212.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.212.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.6.212.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.212.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/HTTP_127.6.212.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.212.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/HTTP_127.6.212.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 11:36:48 dist-test-slave-cq7c krb5kdc[18887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249808, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 11:36:48.597412 6992 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 11:36:48.621847 6992 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 11:36:49.560679 6992 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 11:36:49.560817 6992 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ranger-kms
I20260924 11:36:49.560865 6992 mini_ranger_kms.cc:227] Using postgres at 127.6.212.20:35553
2026-09-24 11:36:49,921 [I] DB FLAVOR :POSTGRES
2026-09-24 11:36:49,921 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 11:36:49,921 [I] Checking connection
2026-09-24 11:36:49,922 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task75Snja/build/asan/bin/postgresql.jar:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.212.20:35553/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 11:36:50,175 [I] connection success
2026-09-24 11:36:50,175 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 11:36:50,175 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 11:36:50,176 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task75Snja/build/asan/bin/postgresql.jar:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.212.20:35553/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 11:36:50,424 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 11:36:50,424 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 11:36:50,424 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 11:36:50,424 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task75Snja/build/asan/bin/postgresql.jar:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.212.20:35553/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 11:36:50,691 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 11:36:51.225505 6992 mini_ranger_kms.cc:310] Created kms service
I20260924 11:36:51.413462 6992 mini_ranger_kms.cc:326] Created kudu user
I20260924 11:36:51.459004 6992 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 11:36:51 dist-test-slave-cq7c krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249808, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.6.212.20@KRBTEST.COM
I20260924 11:36:51.887017 6992 mini_ranger_kms.cc:384] Added ranger policy
I20260924 11:36:51.887401 6992 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ranger-kms:/tmp/dist-test-task75Snja/build/asan/bin/postgresql.jar:/tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task75Snja/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 11:36:51.887458 6992 mini_ranger_kms.cc:240] Using host: 127.6.212.20
I20260924 11:36:51.906955 6992 mini_ranger_kms.cc:276] Ranger KMS PID: 19281
I20260924 11:36:51.907100 6992 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.6.212.20:57751
Sep 24, 2026 11:36:53 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 11:36:53 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 11:36:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 11:36:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 11:36:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 11:36:53 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 11:36:54.281672 6992 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.684s user 0.073s sys 0.306s
I20260924 11:36:54.281908 6992 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 11:36:54.282078 6992 mini_ranger_kms.cc:401] 127.6.212.20:57751/kms/v1/keys
Sep 24 11:36:54 dist-test-slave-cq7c krb5kdc[18887](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790249814, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.212.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 11:36:54 dist-test-slave-cq7c krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249814, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.212.20@KRBTEST.COM for HTTP/127.6.212.20@KRBTEST.COM
Sep 24 11:36:55 dist-test-slave-cq7c krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249814, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.212.20@KRBTEST.COM for HTTP/127.6.212.20@KRBTEST.COM
Sep 24 11:36:57 dist-test-slave-cq7c krb5kdc[18887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.6.212.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
I20260924 11:36:57.768296 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:35281
--webserver_interface=127.6.212.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:34145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:35281
--ranger_config_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.212.20:57751/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:36:58.115737 19331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:36:58.116061 19331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:36:58.116266 19331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:36:58.123860 19331 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:36:58.124002 19331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:36:58.124107 19331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:36:58.124207 19331 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:36:58.124301 19331 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:36:58.134959 19331 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:34145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/master-0/wal
--ranger_kms_url=127.6.212.20:57751/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:35281
--ranger_config_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:35281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.212.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:36:58.136366 19331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:36:58.137787 19331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:36:58.150434 19336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:36:58.150503 19339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:36:58.150466 19337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:36:58.151124 19331 server_base.cc:1034] running on GCE node
I20260924 11:36:58.152096 19331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:36:58.153687 19331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:36:58.154983 19331 hybrid_clock.cc:648] HybridClock initialized: now 1790249818154945 us; error 82 us; skew 500 ppm
I20260924 11:36:58.155519 19331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:36:58 dist-test-slave-cq7c krb5kdc[18887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249818, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:36:58.166000 19331 init.cc:377] Logged in from keytab as kudu/127.6.212.62@KRBTEST.COM (short username kudu)
I20260924 11:36:58.167956 19331 webserver.cc:460] Webserver started at http://127.6.212.62:37575/ using document root <none> and password file <none>
I20260924 11:36:58.168743 19331 fs_manager.cc:359] Metadata directory not provided
I20260924 11:36:58.168905 19331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:36:58.169258 19331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 11:36:58 dist-test-slave-cq7c krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249818, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for HTTP/127.6.212.20@KRBTEST.COM
I20260924 11:36:58.232215 19331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "e3c1a78fe3dd4f0f93b601cbdb13361a"
format_stamp: "Formatted at 2026-09-24 11:36:58 on dist-test-slave-cq7c"
server_key: "0874fbb4aa7ffe2e7c94f2973521e73e"
server_key_iv: "40fcea7618b1a952e34b6078fddb8174"
server_key_version: "kuduclusterkey@0"
I20260924 11:36:58.233208 19331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "e3c1a78fe3dd4f0f93b601cbdb13361a"
format_stamp: "Formatted at 2026-09-24 11:36:58 on dist-test-slave-cq7c"
server_key: "0874fbb4aa7ffe2e7c94f2973521e73e"
server_key_iv: "40fcea7618b1a952e34b6078fddb8174"
server_key_version: "kuduclusterkey@0"
I20260924 11:36:58.238961 19331 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 11:36:58.259325 19346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:36:58.261197 19331 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 11:36:58.261374 19331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "e3c1a78fe3dd4f0f93b601cbdb13361a"
format_stamp: "Formatted at 2026-09-24 11:36:58 on dist-test-slave-cq7c"
server_key: "0874fbb4aa7ffe2e7c94f2973521e73e"
server_key_iv: "40fcea7618b1a952e34b6078fddb8174"
server_key_version: "kuduclusterkey@0"
I20260924 11:36:58.261592 19331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:36:58.276381 19331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:36:58.281008 19331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:36:58.281315 19331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:36:58.302011 19331 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.62:35281
I20260924 11:36:58.302030 19397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.62:35281 every 8 connection(s)
I20260924 11:36:58.303640 19331 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:36:58.304698 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 19331
I20260924 11:36:58.304942 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:36:58.305835 6992 external_mini_cluster.cc:1428] Setting key 225ed19e8055d40456bed8bd1f0bcd14
I20260924 11:36:58.309126 19398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 11:36:58 dist-test-slave-cq7c krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:36:58.324218 19398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3c1a78fe3dd4f0f93b601cbdb13361a: Bootstrap starting.
I20260924 11:36:58.328949 19398 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3c1a78fe3dd4f0f93b601cbdb13361a: Neither blocks nor log segments found. Creating new log.
I20260924 11:36:58.330484 19398 log.cc:824] T 00000000000000000000000000000000 P e3c1a78fe3dd4f0f93b601cbdb13361a: Log is configured to *not* fsync() on all Append() calls
I20260924 11:36:58.332798 19401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:36:58.308135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56154 (local address 127.6.212.62:35281)
0924 11:36:58.308968 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0924 11:36:58.308986 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 11:36:58.309051 (+ 65us) server_negotiation.cc:407] Connection header received
0924 11:36:58.311612 (+ 2561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:36:58.311644 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:36:58.312203 (+ 559us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:36:58.312712 (+ 509us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:36:58.313964 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:36:58.316317 (+ 2353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:36:58.317697 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:36:58.318869 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:36:58.319080 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:36:58.323444 (+ 4364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:36:58.323457 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:36:58.323482 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:36:58.323539 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:36:58.327766 (+ 4227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:36:58.328585 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:36:58.328594 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:36:58.328604 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:36:58.328834 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:36:58.329337 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:36:58.329345 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:36:58.329348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:36:58.330125 (+ 777us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:36:58.330496 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0924 11:36:58.330855 (+ 359us) server_negotiation.cc:299] Negotiation successful
0924 11:36:58.331460 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":367,"threads_started":1}
I20260924 11:36:58.334355 19398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3c1a78fe3dd4f0f93b601cbdb13361a: No bootstrap required, opened a new log
I20260924 11:36:58.343997 19398 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3c1a78fe3dd4f0f93b601cbdb13361a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c1a78fe3dd4f0f93b601cbdb13361a" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 35281 } }
I20260924 11:36:58.344444 19398 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3c1a78fe3dd4f0f93b601cbdb13361a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:36:58.344612 19398 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3c1a78fe3dd4f0f93b601cbdb13361a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c1a78fe3dd4f0f93b601cbdb13361a, State: Initialized, Role: FOLLOWER
I20260924 11:36:58.345312 19398 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3c1a78fe3dd4f0f93b601cbdb13361a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c1a78fe3dd4f0f93b601cbdb13361a" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 35281 } }
I20260924 11:36:58.345587 19398 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3c1a78fe3dd4f0f93b601cbdb13361a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:36:58.345753 19398 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3c1a78fe3dd4f0f93b601cbdb13361a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:36:58.345950 19398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3c1a78fe3dd4f0f93b601cbdb13361a [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:36:58.348424 19398 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3c1a78fe3dd4f0f93b601cbdb13361a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c1a78fe3dd4f0f93b601cbdb13361a" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 35281 } }
I20260924 11:36:58.349023 19398 leader_election.cc:302] T 00000000000000000000000000000000 P e3c1a78fe3dd4f0f93b601cbdb13361a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3c1a78fe3dd4f0f93b601cbdb13361a; no voters:
I20260924 11:36:58.349606 19398 leader_election.cc:288] T 00000000000000000000000000000000 P e3c1a78fe3dd4f0f93b601cbdb13361a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:36:58.349745 19403 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3c1a78fe3dd4f0f93b601cbdb13361a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:36:58.351420 19403 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3c1a78fe3dd4f0f93b601cbdb13361a [term 1 LEADER]: Becoming Leader. State: Replica: e3c1a78fe3dd4f0f93b601cbdb13361a, State: Running, Role: LEADER
I20260924 11:36:58.351948 19403 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3c1a78fe3dd4f0f93b601cbdb13361a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c1a78fe3dd4f0f93b601cbdb13361a" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 35281 } }
I20260924 11:36:58.352440 19398 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3c1a78fe3dd4f0f93b601cbdb13361a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:36:58.354902 19405 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3c1a78fe3dd4f0f93b601cbdb13361a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3c1a78fe3dd4f0f93b601cbdb13361a. Latest consensus state: current_term: 1 leader_uuid: "e3c1a78fe3dd4f0f93b601cbdb13361a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c1a78fe3dd4f0f93b601cbdb13361a" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 35281 } } }
I20260924 11:36:58.355476 19405 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3c1a78fe3dd4f0f93b601cbdb13361a [sys.catalog]: This master's current role is: LEADER
I20260924 11:36:58.355012 19404 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3c1a78fe3dd4f0f93b601cbdb13361a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3c1a78fe3dd4f0f93b601cbdb13361a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c1a78fe3dd4f0f93b601cbdb13361a" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 35281 } } }
I20260924 11:36:58.355700 19404 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3c1a78fe3dd4f0f93b601cbdb13361a [sys.catalog]: This master's current role is: LEADER
I20260924 11:36:58.358520 19412 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:36:58.363029 19412 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:36:58.363901 19398 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 11:36:58.381237 19412 catalog_manager.cc:1347] Generated new cluster ID: 983f34afd7a84ee3a7a8597101072d4e
I20260924 11:36:58.381389 19412 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:36:58.393463 19412 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:36:58.395673 19412 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:36:58.408509 19412 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3c1a78fe3dd4f0f93b601cbdb13361a: Generated new TSK 0
I20260924 11:36:58.409834 19412 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.212.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab.
Entry for principal kudu/127.6.212.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab.
I20260924 11:36:59.184279 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.212.20:57751/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.1:0
--local_ip_for_outbound_sockets=127.6.212.1
--webserver_interface=127.6.212.1
--webserver_port=0
--tserver_master_addrs=127.6.212.62:35281
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab
--principal=kudu/127.6.212.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:34145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 11:36:59 dist-test-slave-cq7c krb5kdc[18887](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790249819, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 11:36:59.634786 19443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:36:59.635221 19443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:36:59.635448 19443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:36:59.647416 19443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:36:59.647579 19443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:36:59.647822 19443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.1
I20260924 11:36:59.664496 19443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:34145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.6.212.20:57751/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab
--principal=kudu/127.6.212.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.212.1
--webserver_port=0
--tserver_master_addrs=127.6.212.62:35281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.1
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:36:59.666375 19443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:36:59.668236 19443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:36:59.693388 19453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:36:59.693400 19454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:36:59.699324 19443 server_base.cc:1034] running on GCE node
W20260924 11:36:59.699815 19456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:36:59.700762 19443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:36:59.702219 19443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:36:59.703531 19443 hybrid_clock.cc:648] HybridClock initialized: now 1790249819703448 us; error 64 us; skew 500 ppm
I20260924 11:36:59.704015 19443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:36:59 dist-test-slave-cq7c krb5kdc[18887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249819, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:36:59.711076 19443 init.cc:377] Logged in from keytab as kudu/127.6.212.1@KRBTEST.COM (short username kudu)
I20260924 11:36:59.714259 19443 webserver.cc:460] Webserver started at http://127.6.212.1:33797/ using document root <none> and password file <none>
I20260924 11:36:59.715130 19443 fs_manager.cc:359] Metadata directory not provided
I20260924 11:36:59.715243 19443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:36:59.715632 19443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 11:36:59 dist-test-slave-cq7c krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249819, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for HTTP/127.6.212.20@KRBTEST.COM
I20260924 11:36:59.735220 19443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-0/data/instance:
uuid: "f1bdad89e3fc460b98a4cd18266ae2ac"
format_stamp: "Formatted at 2026-09-24 11:36:59 on dist-test-slave-cq7c"
server_key: "2eb9c1b87994fb98e6ff7596ef138ead"
server_key_iv: "3e4c9ba8321c8db29f66e77a8b51ec9d"
server_key_version: "kuduclusterkey@0"
I20260924 11:36:59.736645 19443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance:
uuid: "f1bdad89e3fc460b98a4cd18266ae2ac"
format_stamp: "Formatted at 2026-09-24 11:36:59 on dist-test-slave-cq7c"
server_key: "2eb9c1b87994fb98e6ff7596ef138ead"
server_key_iv: "3e4c9ba8321c8db29f66e77a8b51ec9d"
server_key_version: "kuduclusterkey@0"
I20260924 11:36:59.752453 19443 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.010s sys 0.005s
I20260924 11:36:59.775734 19463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:36:59.779419 19443 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.001s
I20260924 11:36:59.779691 19443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-0/wal
uuid: "f1bdad89e3fc460b98a4cd18266ae2ac"
format_stamp: "Formatted at 2026-09-24 11:36:59 on dist-test-slave-cq7c"
server_key: "2eb9c1b87994fb98e6ff7596ef138ead"
server_key_iv: "3e4c9ba8321c8db29f66e77a8b51ec9d"
server_key_version: "kuduclusterkey@0"
I20260924 11:36:59.780119 19443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:36:59.851900 19443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:36:59.857251 19443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:36:59.857712 19443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:36:59.861487 19443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:36:59.863348 19443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:36:59.863513 19443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260924 11:36:59.863652 19443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:36:59.863734 19443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 11:36:59 dist-test-slave-cq7c krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249819, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for HTTP/127.6.212.20@KRBTEST.COM
I20260924 11:37:00.091777 19443 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.1:35781
I20260924 11:37:00.091770 19575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.1:35781 every 8 connection(s)
I20260924 11:37:00.093665 19443 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
I20260924 11:37:00.097926 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 19443
I20260924 11:37:00.098224 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance
I20260924 11:37:00.099128 6992 external_mini_cluster.cc:1428] Setting key 0493eb9253bed1b2ccd55fbcc539a487
Sep 24 11:37:00 dist-test-slave-cq7c krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249819, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.212.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.2@KRBTEST.COM" created.
I20260924 11:37:00.161129 19579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:00.098463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:47215 (local address 127.6.212.62:35281)
0924 11:37:00.117801 (+ 19338us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:00.117811 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:00.119538 (+ 1727us) server_negotiation.cc:407] Connection header received
0924 11:37:00.122110 (+ 2572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:00.122117 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:00.122216 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:00.122331 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:00.124871 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:00.126082 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:00.128828 (+ 2746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:00.129601 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:00.129721 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:00.135710 (+ 5989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:00.135719 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:00.135725 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:00.135776 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:00.139813 (+ 4037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:00.147354 (+ 7541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:00.147391 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:00.147395 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:00.147630 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:00.148510 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:00.148516 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:00.148519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:00.148910 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:00.149055 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:00.160375 (+ 11320us) server_negotiation.cc:299] Negotiation successful
0924 11:37:00.160710 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19155,"thread_start_us":18985,"threads_started":1}
I20260924 11:37:00.164194 19578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:00.097679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35281 (local address 127.6.212.1:47215)
0924 11:37:00.118588 (+ 20909us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:00.118665 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0924 11:37:00.121478 (+ 2813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:37:00.122872 (+ 1394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:37:00.122887 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:37:00.123654 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:37:00.124603 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:00.124639 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:00.126323 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:00.126330 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:00.128620 (+ 2290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:00.128639 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:00.130121 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:00.130129 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:00.130503 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:00.132198 (+ 1695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:37:00.132247 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:37:00.135351 (+ 3104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:37:00.140043 (+ 4692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:00.140056 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:00.140078 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:00.140828 (+ 750us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:00.147851 (+ 7023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:00.147857 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:00.147861 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:00.148179 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:00.149168 (+ 989us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:37:00.149177 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:37:00.149758 (+ 581us) client_negotiation.cc:769] Sending connection context
0924 11:37:00.161733 (+ 11975us) client_negotiation.cc:240] Negotiation successful
0924 11:37:00.162449 (+ 716us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20446,"thread_start_us":33295,"threads_started":1}
I20260924 11:37:00.166579 19576 heartbeater.cc:347] Connected to a master server at 127.6.212.62:35281
I20260924 11:37:00.166936 19576 heartbeater.cc:464] Registering TS with master...
I20260924 11:37:00.167802 19576 heartbeater.cc:511] Master 127.6.212.62:35281 requested a full tablet report, sending...
I20260924 11:37:00.170940 19363 ts_manager.cc:194] Registered new tserver with Master: f1bdad89e3fc460b98a4cd18266ae2ac (127.6.212.1:35781)
I20260924 11:37:00.173210 19363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.1@KRBTEST.COM'} at 127.6.212.1:47215
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab.
Entry for principal kudu/127.6.212.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab.
I20260924 11:37:00.220357 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.212.20:57751/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.2:0
--local_ip_for_outbound_sockets=127.6.212.2
--webserver_interface=127.6.212.2
--webserver_port=0
--tserver_master_addrs=127.6.212.62:35281
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab
--principal=kudu/127.6.212.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:34145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 11:37:00 dist-test-slave-cq7c krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249819, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for HTTP/127.6.212.20@KRBTEST.COM
W20260924 11:37:00.753455 19582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:37:00.753803 19582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:37:00.753952 19582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:37:00.761817 19582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:37:00.761950 19582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:37:00.762130 19582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.2
I20260924 11:37:00.772969 19582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:34145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.6.212.20:57751/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab
--principal=kudu/127.6.212.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.212.2
--webserver_port=0
--tserver_master_addrs=127.6.212.62:35281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.2
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:37:00.774397 19582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:37:00.775863 19582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:37:00.791083 19597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:37:00.791388 19596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:37:00.791736 19599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:37:00.792570 19582 server_base.cc:1034] running on GCE node
I20260924 11:37:00.793355 19582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:37:00.794620 19582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:37:00.795871 19582 hybrid_clock.cc:648] HybridClock initialized: now 1790249820795792 us; error 71 us; skew 500 ppm
I20260924 11:37:00.796262 19582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:37:00 dist-test-slave-cq7c krb5kdc[18887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249820, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:37:00.800297 19582 init.cc:377] Logged in from keytab as kudu/127.6.212.2@KRBTEST.COM (short username kudu)
I20260924 11:37:00.802399 19582 webserver.cc:460] Webserver started at http://127.6.212.2:38463/ using document root <none> and password file <none>
I20260924 11:37:00.803309 19582 fs_manager.cc:359] Metadata directory not provided
I20260924 11:37:00.803488 19582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:37:00.803819 19582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 11:37:00 dist-test-slave-cq7c krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249820, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.2@KRBTEST.COM for HTTP/127.6.212.20@KRBTEST.COM
I20260924 11:37:00.821723 19582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-1/data/instance:
uuid: "14cb8e071a7a4fce8653d37580a207b6"
format_stamp: "Formatted at 2026-09-24 11:37:00 on dist-test-slave-cq7c"
server_key: "3d0412d3f065b91aea084e2226205ba6"
server_key_iv: "4dd8c3e645824fbfc0c48d364df2d428"
server_key_version: "kuduclusterkey@0"
I20260924 11:37:00.823007 19582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance:
uuid: "14cb8e071a7a4fce8653d37580a207b6"
format_stamp: "Formatted at 2026-09-24 11:37:00 on dist-test-slave-cq7c"
server_key: "3d0412d3f065b91aea084e2226205ba6"
server_key_iv: "4dd8c3e645824fbfc0c48d364df2d428"
server_key_version: "kuduclusterkey@0"
I20260924 11:37:00.830662 19582 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.005s
I20260924 11:37:00.852120 19606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:00.854370 19582 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260924 11:37:00.854612 19582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-1/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-1/wal
uuid: "14cb8e071a7a4fce8653d37580a207b6"
format_stamp: "Formatted at 2026-09-24 11:37:00 on dist-test-slave-cq7c"
server_key: "3d0412d3f065b91aea084e2226205ba6"
server_key_iv: "4dd8c3e645824fbfc0c48d364df2d428"
server_key_version: "kuduclusterkey@0"
I20260924 11:37:00.854887 19582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:37:00.883464 19582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:37:00.889698 19582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:37:00.890090 19582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:37:00.891601 19582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:37:00.894315 19582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:37:00.894531 19582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:00.894866 19582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:37:00.895000 19582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:00.943885 19582 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.2:36961
I20260924 11:37:00.943898 19718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.2:36961 every 8 connection(s)
I20260924 11:37:00.945521 19582 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
I20260924 11:37:00.946002 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 19582
I20260924 11:37:00.946193 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance
I20260924 11:37:00.947175 6992 external_mini_cluster.cc:1428] Setting key 172e38f9da4f9330c02264080c0a718c
Sep 24 11:37:00 dist-test-slave-cq7c krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249820, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.2@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:37:00.989208 19721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:00.951761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:40153 (local address 127.6.212.62:35281)
0924 11:37:00.967510 (+ 15749us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:00.967517 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:00.967539 (+ 22us) server_negotiation.cc:407] Connection header received
0924 11:37:00.967601 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:00.967606 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:00.967698 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:00.967803 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:00.969994 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:00.971061 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:00.973918 (+ 2857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:00.974537 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:00.974673 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:00.979672 (+ 4999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:00.979680 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:00.979685 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:00.979733 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:00.985161 (+ 5428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:00.986322 (+ 1161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:00.986327 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:00.986331 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:00.986489 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:00.987040 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:00.987045 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:00.987048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:00.987462 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:00.987610 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:00.988575 (+ 965us) server_negotiation.cc:299] Negotiation successful
0924 11:37:00.988820 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15568,"thread_start_us":19014,"threads_started":1}
I20260924 11:37:00.990597 19722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:00.952474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35281 (local address 127.6.212.2:40153)
0924 11:37:00.964091 (+ 11617us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:00.964154 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 11:37:00.966731 (+ 2577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:37:00.968256 (+ 1525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:37:00.968270 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:37:00.968902 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:37:00.969763 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:00.969788 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:00.971299 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:00.971310 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:00.973728 (+ 2418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:00.973749 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:00.974837 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:00.974846 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:00.975181 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:00.976128 (+ 947us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:37:00.976165 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:37:00.979219 (+ 3054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:37:00.985371 (+ 6152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:00.985384 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:00.985404 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:00.986136 (+ 732us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:00.986597 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:00.986604 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:00.986608 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:00.986887 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:00.987594 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:37:00.987604 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:37:00.988064 (+ 460us) client_negotiation.cc:769] Sending connection context
0924 11:37:00.988438 (+ 374us) client_negotiation.cc:240] Negotiation successful
0924 11:37:00.989042 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11221,"thread_start_us":12697,"threads_started":1}
I20260924 11:37:00.993175 19719 heartbeater.cc:347] Connected to a master server at 127.6.212.62:35281
I20260924 11:37:00.993543 19719 heartbeater.cc:464] Registering TS with master...
I20260924 11:37:00.994459 19719 heartbeater.cc:511] Master 127.6.212.62:35281 requested a full tablet report, sending...
I20260924 11:37:00.996384 19363 ts_manager.cc:194] Registered new tserver with Master: 14cb8e071a7a4fce8653d37580a207b6 (127.6.212.2:36961)
I20260924 11:37:00.997530 19363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.2@KRBTEST.COM'} at 127.6.212.2:40153
WARNING: no policy specified for kudu/127.6.212.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab.
Entry for principal kudu/127.6.212.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab.
I20260924 11:37:01.043246 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.212.20:57751/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.3:0
--local_ip_for_outbound_sockets=127.6.212.3
--webserver_interface=127.6.212.3
--webserver_port=0
--tserver_master_addrs=127.6.212.62:35281
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab
--principal=kudu/127.6.212.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:34145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:37:01.178004 19576 heartbeater.cc:503] Master 127.6.212.62:35281 was elected leader, sending a full tablet report...
W20260924 11:37:01.424786 19725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:37:01.425130 19725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:37:01.425313 19725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:37:01.432928 19725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:37:01.433079 19725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:37:01.433267 19725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.3
I20260924 11:37:01.443377 19725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:34145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.6.212.20:57751/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab
--principal=kudu/127.6.212.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.212.3
--webserver_port=0
--tserver_master_addrs=127.6.212.62:35281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.3
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:37:01.444744 19725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:37:01.446151 19725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:37:01.460263 19731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:37:01.460382 19733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:37:01.460409 19730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:37:01.461310 19725 server_base.cc:1034] running on GCE node
I20260924 11:37:01.462055 19725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:37:01.463169 19725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:37:01.464418 19725 hybrid_clock.cc:648] HybridClock initialized: now 1790249821464339 us; error 82 us; skew 500 ppm
I20260924 11:37:01.464802 19725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:37:01 dist-test-slave-cq7c krb5kdc[18887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249821, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:37:01.468415 19725 init.cc:377] Logged in from keytab as kudu/127.6.212.3@KRBTEST.COM (short username kudu)
I20260924 11:37:01.470052 19725 webserver.cc:460] Webserver started at http://127.6.212.3:32859/ using document root <none> and password file <none>
I20260924 11:37:01.470736 19725 fs_manager.cc:359] Metadata directory not provided
I20260924 11:37:01.470836 19725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:37:01.471099 19725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 11:37:01 dist-test-slave-cq7c krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249821, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.3@KRBTEST.COM for HTTP/127.6.212.20@KRBTEST.COM
I20260924 11:37:01.486835 19725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-2/data/instance:
uuid: "865b55a6de8743199673497e5298532e"
format_stamp: "Formatted at 2026-09-24 11:37:01 on dist-test-slave-cq7c"
server_key: "c9c6e99658503b96b3b00ff414643849"
server_key_iv: "9e961ec706aca7a7af6d58433dde7977"
server_key_version: "kuduclusterkey@0"
I20260924 11:37:01.487804 19725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance:
uuid: "865b55a6de8743199673497e5298532e"
format_stamp: "Formatted at 2026-09-24 11:37:01 on dist-test-slave-cq7c"
server_key: "c9c6e99658503b96b3b00ff414643849"
server_key_iv: "9e961ec706aca7a7af6d58433dde7977"
server_key_version: "kuduclusterkey@0"
I20260924 11:37:01.493974 19725 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 11:37:01.510612 19740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:01.512426 19725 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260924 11:37:01.512593 19725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-2/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-2/wal
uuid: "865b55a6de8743199673497e5298532e"
format_stamp: "Formatted at 2026-09-24 11:37:01 on dist-test-slave-cq7c"
server_key: "c9c6e99658503b96b3b00ff414643849"
server_key_iv: "9e961ec706aca7a7af6d58433dde7977"
server_key_version: "kuduclusterkey@0"
I20260924 11:37:01.512805 19725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:37:01.531983 19725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:37:01.535863 19725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:37:01.536146 19725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:37:01.537075 19725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:37:01.538491 19725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:37:01.538563 19725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:01.538650 19725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:37:01.538692 19725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:01.571996 19725 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.3:41513
I20260924 11:37:01.572031 19852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.3:41513 every 8 connection(s)
I20260924 11:37:01.573407 19725 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
I20260924 11:37:01.575738 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 19725
I20260924 11:37:01.575928 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance
I20260924 11:37:01.576642 6992 external_mini_cluster.cc:1428] Setting key e3ecc3bc727a11bc999a25de3e4e1263
Sep 24 11:37:01 dist-test-slave-cq7c krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249821, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.3@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:37:01.595551 19855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:01.576132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:50779 (local address 127.6.212.62:35281)
0924 11:37:01.576608 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:01.576614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:01.577834 (+ 1220us) server_negotiation.cc:407] Connection header received
0924 11:37:01.580281 (+ 2447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:01.580287 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:01.580400 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:01.580520 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:01.582084 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:01.582804 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:01.584519 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:01.584977 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:01.585124 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:01.588734 (+ 3610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:01.588747 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:01.588751 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:01.588796 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:01.592030 (+ 3234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:01.592982 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:01.592987 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:01.592989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:01.593123 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:01.593683 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:01.593687 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:01.593689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:01.594027 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:01.594153 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:01.594891 (+ 738us) server_negotiation.cc:299] Negotiation successful
0924 11:37:01.595185 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":230,"threads_started":1}
I20260924 11:37:01.596405 19856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:01.576786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35281 (local address 127.6.212.3:50779)
0924 11:37:01.577663 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:01.577720 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 11:37:01.579954 (+ 2234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:37:01.580800 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:37:01.580808 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:37:01.581316 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:37:01.581913 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:01.581931 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:01.582971 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:01.582977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:01.584376 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:01.584388 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:01.585229 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:01.585236 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:01.585500 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:01.586237 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:37:01.586260 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:37:01.588515 (+ 2255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:37:01.592165 (+ 3650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:01.592175 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:01.592194 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:01.592804 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:01.593310 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:01.593314 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:01.593316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:01.593553 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:01.594156 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:37:01.594162 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:37:01.594494 (+ 332us) client_negotiation.cc:769] Sending connection context
0924 11:37:01.594753 (+ 259us) client_negotiation.cc:240] Negotiation successful
0924 11:37:01.595173 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":377,"threads_started":1}
I20260924 11:37:01.597828 19853 heartbeater.cc:347] Connected to a master server at 127.6.212.62:35281
I20260924 11:37:01.598176 19853 heartbeater.cc:464] Registering TS with master...
I20260924 11:37:01.598953 19853 heartbeater.cc:511] Master 127.6.212.62:35281 requested a full tablet report, sending...
I20260924 11:37:01.600824 19363 ts_manager.cc:194] Registered new tserver with Master: 865b55a6de8743199673497e5298532e (127.6.212.3:41513)
I20260924 11:37:01.601747 19363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.3@KRBTEST.COM'} at 127.6.212.3:50779
I20260924 11:37:01.606436 6992 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:37:01.608263 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 19443
Sep 24 11:37:01 dist-test-slave-cq7c krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.1@KRBTEST.COM
I20260924 11:37:01.626091 19858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:01.609364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42270 (local address 127.6.212.1:35781)
0924 11:37:01.610010 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:01.610019 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:01.610048 (+ 29us) server_negotiation.cc:407] Connection header received
0924 11:37:01.610107 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:01.610114 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:01.610331 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:01.610476 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:01.612083 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:01.613156 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:01.614602 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:01.615149 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:01.615248 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:01.620423 (+ 5175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:01.620438 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:01.620453 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:01.620520 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:01.623488 (+ 2968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:01.624207 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:01.624215 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:01.624217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:01.624362 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:01.624702 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:01.624705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:01.624707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:01.625043 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:01.625277 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:01.625580 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 11:37:01.625790 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":332,"threads_started":1}
I20260924 11:37:01.627485 19571 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:37:01.716506 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 19443
I20260924 11:37:01.733229 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 19582
Sep 24 11:37:01 dist-test-slave-cq7c krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.2@KRBTEST.COM
I20260924 11:37:01.755631 19860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:01.734311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55560 (local address 127.6.212.2:36961)
0924 11:37:01.734921 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:01.734930 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:01.734954 (+ 24us) server_negotiation.cc:407] Connection header received
0924 11:37:01.735015 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:01.735022 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:01.735268 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:01.735470 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:01.736806 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:01.737819 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:01.739236 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:01.739867 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:01.740019 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:01.749145 (+ 9126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:01.749163 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:01.749180 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:01.749222 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:01.752359 (+ 3137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:01.753337 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:01.753345 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:01.753348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:01.753488 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:01.753898 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:01.753901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:01.753903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:01.754247 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:01.754459 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:01.754811 (+ 352us) server_negotiation.cc:299] Negotiation successful
0924 11:37:01.755052 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":350,"threads_started":1}
I20260924 11:37:01.757136 19714 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:37:01.842458 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 19582
I20260924 11:37:01.860014 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 19725
Sep 24 11:37:01 dist-test-slave-cq7c krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.3@KRBTEST.COM
I20260924 11:37:01.878033 19862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:01.861182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38058 (local address 127.6.212.3:41513)
0924 11:37:01.861791 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:01.861798 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:01.861828 (+ 30us) server_negotiation.cc:407] Connection header received
0924 11:37:01.861888 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:01.861894 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:01.862138 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:01.862269 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:01.863616 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:01.864721 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:01.866224 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:01.866856 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:01.866969 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:01.871740 (+ 4771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:01.871755 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:01.871770 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:01.871822 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:01.874941 (+ 3119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:01.875899 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:01.875906 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:01.875909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:01.876095 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:01.876504 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:01.876507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:01.876509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:01.876868 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:01.877149 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:01.877394 (+ 245us) server_negotiation.cc:299] Negotiation successful
0924 11:37:01.877611 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":312,"threads_started":1}
I20260924 11:37:01.879633 19848 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:37:01.964682 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 19725
I20260924 11:37:01.980553 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 19331
I20260924 11:37:01.981884 19393 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:37:02.090454 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 19331
I20260924 11:37:02.117821 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:35281
--webserver_interface=127.6.212.62
--webserver_port=37575
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:34145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:35281
--ranger_config_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.212.20:57751/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.212.20:57751/kms with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:37:02.455256 19865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:37:02.455719 19865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:37:02.455878 19865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:37:02.463479 19865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:37:02.463647 19865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:37:02.463727 19865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:37:02.463825 19865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:37:02.463896 19865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:37:02.474493 19865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:34145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.212.20:57751/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:35281
--ranger_config_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:35281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.212.62
--webserver_port=37575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:37:02.475915 19865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:37:02.477305 19865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:37:02.489920 19870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:37:02.489876 19873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:37:02.489827 19871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:37:02.491291 19865 server_base.cc:1034] running on GCE node
I20260924 11:37:02.492272 19865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:37:02.493700 19865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:37:02.494915 19865 hybrid_clock.cc:648] HybridClock initialized: now 1790249822494871 us; error 27 us; skew 500 ppm
I20260924 11:37:02.495344 19865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:37:02 dist-test-slave-cq7c krb5kdc[18887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249822, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:37:02.498656 19865 init.cc:377] Logged in from keytab as kudu/127.6.212.62@KRBTEST.COM (short username kudu)
I20260924 11:37:02.500232 19865 webserver.cc:460] Webserver started at http://127.6.212.62:37575/ using document root <none> and password file <none>
I20260924 11:37:02.500888 19865 fs_manager.cc:359] Metadata directory not provided
I20260924 11:37:02.500993 19865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 11:37:02 dist-test-slave-cq7c krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249822, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for HTTP/127.6.212.20@KRBTEST.COM
I20260924 11:37:02.571858 19865 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 11:37:02.575968 19881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:02.577817 19865 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 11:37:02.578045 19865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "e3c1a78fe3dd4f0f93b601cbdb13361a"
format_stamp: "Formatted at 2026-09-24 11:36:58 on dist-test-slave-cq7c"
server_key: "0874fbb4aa7ffe2e7c94f2973521e73e"
server_key_iv: "40fcea7618b1a952e34b6078fddb8174"
server_key_version: "kuduclusterkey@0"
I20260924 11:37:02.579185 19865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:37:02.613694 19865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:37:02.617750 19865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:37:02.618064 19865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:37:02.637445 19865 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.62:35281
I20260924 11:37:02.637470 19932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.62:35281 every 8 connection(s)
I20260924 11:37:02.639024 19865 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:37:02.643158 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 19865
I20260924 11:37:02.644214 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.212.20:57751/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.1:35781
--local_ip_for_outbound_sockets=127.6.212.1
--tserver_master_addrs=127.6.212.62:35281
--webserver_port=33797
--webserver_interface=127.6.212.1
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab
--principal=kudu/127.6.212.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:34145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.212.20:57751/kms with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:37:02.647928 19933 sys_catalog.cc:263] Verifying existing consensus state
I20260924 11:37:02.653283 19933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3c1a78fe3dd4f0f93b601cbdb13361a: Bootstrap starting.
I20260924 11:37:02.672691 19933 log.cc:824] T 00000000000000000000000000000000 P e3c1a78fe3dd4f0f93b601cbdb13361a: Log is configured to *not* fsync() on all Append() calls
I20260924 11:37:02.681744 19933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3c1a78fe3dd4f0f93b601cbdb13361a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 11:37:02.682293 19933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3c1a78fe3dd4f0f93b601cbdb13361a: Bootstrap complete.
I20260924 11:37:02.691671 19933 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3c1a78fe3dd4f0f93b601cbdb13361a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c1a78fe3dd4f0f93b601cbdb13361a" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 35281 } }
I20260924 11:37:02.692842 19933 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3c1a78fe3dd4f0f93b601cbdb13361a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c1a78fe3dd4f0f93b601cbdb13361a, State: Initialized, Role: FOLLOWER
I20260924 11:37:02.693547 19933 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3c1a78fe3dd4f0f93b601cbdb13361a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c1a78fe3dd4f0f93b601cbdb13361a" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 35281 } }
I20260924 11:37:02.693782 19933 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3c1a78fe3dd4f0f93b601cbdb13361a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:37:02.693899 19933 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3c1a78fe3dd4f0f93b601cbdb13361a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:37:02.694095 19933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3c1a78fe3dd4f0f93b601cbdb13361a [term 1 FOLLOWER]: Advancing to term 2
I20260924 11:37:02.698468 19933 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3c1a78fe3dd4f0f93b601cbdb13361a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c1a78fe3dd4f0f93b601cbdb13361a" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 35281 } }
I20260924 11:37:02.699013 19933 leader_election.cc:302] T 00000000000000000000000000000000 P e3c1a78fe3dd4f0f93b601cbdb13361a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3c1a78fe3dd4f0f93b601cbdb13361a; no voters:
I20260924 11:37:02.699780 19933 leader_election.cc:288] T 00000000000000000000000000000000 P e3c1a78fe3dd4f0f93b601cbdb13361a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 11:37:02.699911 19937 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3c1a78fe3dd4f0f93b601cbdb13361a [term 2 FOLLOWER]: Leader election won for term 2
I20260924 11:37:02.701304 19937 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3c1a78fe3dd4f0f93b601cbdb13361a [term 2 LEADER]: Becoming Leader. State: Replica: e3c1a78fe3dd4f0f93b601cbdb13361a, State: Running, Role: LEADER
I20260924 11:37:02.701870 19937 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3c1a78fe3dd4f0f93b601cbdb13361a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c1a78fe3dd4f0f93b601cbdb13361a" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 35281 } }
I20260924 11:37:02.702343 19933 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3c1a78fe3dd4f0f93b601cbdb13361a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:37:02.704721 19939 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3c1a78fe3dd4f0f93b601cbdb13361a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3c1a78fe3dd4f0f93b601cbdb13361a. Latest consensus state: current_term: 2 leader_uuid: "e3c1a78fe3dd4f0f93b601cbdb13361a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c1a78fe3dd4f0f93b601cbdb13361a" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 35281 } } }
I20260924 11:37:02.705061 19939 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3c1a78fe3dd4f0f93b601cbdb13361a [sys.catalog]: This master's current role is: LEADER
I20260924 11:37:02.705828 19938 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3c1a78fe3dd4f0f93b601cbdb13361a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e3c1a78fe3dd4f0f93b601cbdb13361a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c1a78fe3dd4f0f93b601cbdb13361a" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 35281 } } }
I20260924 11:37:02.706281 19938 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3c1a78fe3dd4f0f93b601cbdb13361a [sys.catalog]: This master's current role is: LEADER
I20260924 11:37:02.720395 19942 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:37:02.730038 19942 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:37:02.731534 19942 catalog_manager.cc:1259] Loaded cluster ID: 983f34afd7a84ee3a7a8597101072d4e
I20260924 11:37:02.731726 19942 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:37:02.735970 19942 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:37:02.737969 19942 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e3c1a78fe3dd4f0f93b601cbdb13361a: Loaded TSK: 0
I20260924 11:37:02.739917 19942 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:37:02.760607 19933 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260924 11:37:03.109200 19935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:37:03.109640 19935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:37:03.109824 19935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:37:03.122419 19935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:37:03.122574 19935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:37:03.122772 19935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.1
I20260924 11:37:03.139717 19935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:34145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.212.20:57751/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab
--principal=kudu/127.6.212.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.1:35781
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.212.1
--webserver_port=33797
--tserver_master_addrs=127.6.212.62:35281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.1
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:37:03.141587 19935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:37:03.143721 19935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:37:03.171909 19970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:37:03.174188 19971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:37:03.178989 19935 server_base.cc:1034] running on GCE node
W20260924 11:37:03.179811 19973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:37:03.180706 19935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:37:03.182121 19935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:37:03.183419 19935 hybrid_clock.cc:648] HybridClock initialized: now 1790249823183307 us; error 67 us; skew 500 ppm
I20260924 11:37:03.183880 19935 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:37:03 dist-test-slave-cq7c krb5kdc[18887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249823, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:37:03.197515 19935 init.cc:377] Logged in from keytab as kudu/127.6.212.1@KRBTEST.COM (short username kudu)
I20260924 11:37:03.203504 19935 webserver.cc:460] Webserver started at http://127.6.212.1:33797/ using document root <none> and password file <none>
I20260924 11:37:03.204545 19935 fs_manager.cc:359] Metadata directory not provided
I20260924 11:37:03.204720 19935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 11:37:03 dist-test-slave-cq7c krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249823, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for HTTP/127.6.212.20@KRBTEST.COM
I20260924 11:37:03.245889 19935 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.003s sys 0.003s
I20260924 11:37:03.249571 19981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:03.251263 19935 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 11:37:03.251523 19935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-0/wal
uuid: "f1bdad89e3fc460b98a4cd18266ae2ac"
format_stamp: "Formatted at 2026-09-24 11:36:59 on dist-test-slave-cq7c"
server_key: "2eb9c1b87994fb98e6ff7596ef138ead"
server_key_iv: "3e4c9ba8321c8db29f66e77a8b51ec9d"
server_key_version: "kuduclusterkey@0"
I20260924 11:37:03.252744 19935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:37:03.321483 19935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:37:03.327015 19935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:37:03.327315 19935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:37:03.334772 19935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:37:03.343019 19935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:37:03.343165 19935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:03.343302 19935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:37:03.343400 19935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:03.427804 19935 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.1:35781
I20260924 11:37:03.427882 20094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.1:35781 every 8 connection(s)
I20260924 11:37:03.430171 19935 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
I20260924 11:37:03.437836 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 19935
I20260924 11:37:03.438805 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.212.20:57751/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.2:36961
--local_ip_for_outbound_sockets=127.6.212.2
--tserver_master_addrs=127.6.212.62:35281
--webserver_port=38463
--webserver_interface=127.6.212.2
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab
--principal=kudu/127.6.212.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:34145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.212.20:57751/kms with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 11:37:03 dist-test-slave-cq7c krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249823, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:37:03.475550 20097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:03.434194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35281 (local address 127.6.212.1:44411)
0924 11:37:03.438129 (+ 3935us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:03.438194 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 11:37:03.441475 (+ 3281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:37:03.444703 (+ 3228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:37:03.444711 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:37:03.445396 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:37:03.447915 (+ 2519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:03.447953 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:03.451111 (+ 3158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:03.451120 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:03.453951 (+ 2831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:03.453999 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:03.456481 (+ 2482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:03.456492 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:03.456871 (+ 379us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:03.457978 (+ 1107us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:37:03.458023 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:37:03.463243 (+ 5220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:37:03.468710 (+ 5467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:03.468727 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:03.468754 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:03.469664 (+ 910us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:03.470342 (+ 678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:03.470351 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:03.470358 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:03.470746 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:03.471964 (+ 1218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:37:03.471976 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:37:03.472823 (+ 847us) client_negotiation.cc:769] Sending connection context
0924 11:37:03.473210 (+ 387us) client_negotiation.cc:240] Negotiation successful
0924 11:37:03.473897 (+ 687us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3558,"thread_start_us":7650,"threads_started":1}
I20260924 11:37:03.478644 20098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:03.435115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:44411 (local address 127.6.212.62:35281)
0924 11:37:03.437437 (+ 2322us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:03.437461 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:03.438371 (+ 910us) server_negotiation.cc:407] Connection header received
0924 11:37:03.442364 (+ 3993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:03.442386 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:03.442932 (+ 546us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:03.443789 (+ 857us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:03.448290 (+ 4501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:03.450837 (+ 2547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:03.454235 (+ 3398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:03.455702 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:03.455988 (+ 286us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:03.463796 (+ 7808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:03.463809 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:03.463836 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:03.463898 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:03.468438 (+ 4540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:03.469899 (+ 1461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:03.469911 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:03.469920 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:03.470157 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:03.470918 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:03.470925 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:03.470930 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:03.471701 (+ 771us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:03.471899 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:03.476590 (+ 4691us) server_negotiation.cc:299] Negotiation successful
0924 11:37:03.477091 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1970,"thread_start_us":1616,"threads_started":1}
I20260924 11:37:03.482507 20095 heartbeater.cc:347] Connected to a master server at 127.6.212.62:35281
I20260924 11:37:03.482854 20095 heartbeater.cc:464] Registering TS with master...
I20260924 11:37:03.484023 20095 heartbeater.cc:511] Master 127.6.212.62:35281 requested a full tablet report, sending...
W20260924 11:37:03.487712 20095 heartbeater.cc:644] Failed to heartbeat to 127.6.212.62:35281 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 11:37:03.487977 20095 heartbeater.cc:464] Registering TS with master...
I20260924 11:37:03.488299 20095 heartbeater.cc:511] Master 127.6.212.62:35281 requested a full tablet report, sending...
I20260924 11:37:03.489601 20095 heartbeater.cc:464] Registering TS with master...
I20260924 11:37:03.489882 20095 heartbeater.cc:511] Master 127.6.212.62:35281 requested a full tablet report, sending...
W20260924 11:37:03.491084 20095 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 11:37:03.951319 20099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:37:03.951772 20099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:37:03.951926 20099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:37:03.963837 20099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:37:03.964000 20099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:37:03.964193 20099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.2
I20260924 11:37:03.980614 20099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:34145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.212.20:57751/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab
--principal=kudu/127.6.212.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.2:36961
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.212.2
--webserver_port=38463
--tserver_master_addrs=127.6.212.62:35281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.2
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:37:03.982590 20099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:37:03.984532 20099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:37:04.002470 20113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:37:04.002908 20114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:37:04.003842 20116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:37:04.005435 20099 server_base.cc:1034] running on GCE node
I20260924 11:37:04.006285 20099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:37:04.007628 20099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:37:04.008889 20099 hybrid_clock.cc:648] HybridClock initialized: now 1790249824008824 us; error 46 us; skew 500 ppm
I20260924 11:37:04.009313 20099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:37:04 dist-test-slave-cq7c krb5kdc[18887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249824, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:37:04.013751 20099 init.cc:377] Logged in from keytab as kudu/127.6.212.2@KRBTEST.COM (short username kudu)
I20260924 11:37:04.015796 20099 webserver.cc:460] Webserver started at http://127.6.212.2:38463/ using document root <none> and password file <none>
I20260924 11:37:04.016819 20099 fs_manager.cc:359] Metadata directory not provided
I20260924 11:37:04.017004 20099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 11:37:04 dist-test-slave-cq7c krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249824, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.2@KRBTEST.COM for HTTP/127.6.212.20@KRBTEST.COM
I20260924 11:37:04.039309 20099 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260924 11:37:04.044503 20124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:04.048867 20099 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.002s
I20260924 11:37:04.049198 20099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-1/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-1/wal
uuid: "14cb8e071a7a4fce8653d37580a207b6"
format_stamp: "Formatted at 2026-09-24 11:37:00 on dist-test-slave-cq7c"
server_key: "3d0412d3f065b91aea084e2226205ba6"
server_key_iv: "4dd8c3e645824fbfc0c48d364df2d428"
server_key_version: "kuduclusterkey@0"
I20260924 11:37:04.050803 20099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:37:04.083817 20099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:37:04.089874 20099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:37:04.090268 20099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:37:04.099449 20099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:37:04.104300 20099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:37:04.104449 20099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:04.104574 20099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:37:04.104666 20099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:04.151144 20099 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.2:36961
I20260924 11:37:04.151247 20238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.2:36961 every 8 connection(s)
I20260924 11:37:04.152797 20099 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
I20260924 11:37:04.162007 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 20099
I20260924 11:37:04.162804 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.212.20:57751/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.3:41513
--local_ip_for_outbound_sockets=127.6.212.3
--tserver_master_addrs=127.6.212.62:35281
--webserver_port=32859
--webserver_interface=127.6.212.3
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab
--principal=kudu/127.6.212.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:34145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.212.20:57751/kms with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 11:37:04 dist-test-slave-cq7c krb5kdc[18887](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790249824, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 11:37:04 dist-test-slave-cq7c krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249824, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.2@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:37:04.187033 20242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:04.156494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35281 (local address 127.6.212.2:33079)
0924 11:37:04.157390 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:04.157452 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 11:37:04.160080 (+ 2628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:37:04.160948 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:37:04.160961 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:37:04.161604 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:37:04.162493 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:04.162522 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:04.164659 (+ 2137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:04.164665 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:04.166876 (+ 2211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:04.166897 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:04.168550 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:04.168561 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:04.168965 (+ 404us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:04.169873 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:37:04.169924 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:37:04.172716 (+ 2792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:37:04.180955 (+ 8239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:04.180967 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:04.180990 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:04.181734 (+ 744us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:04.182378 (+ 644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:04.182383 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:04.182388 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:04.182666 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:04.183653 (+ 987us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:37:04.183663 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:37:04.184202 (+ 539us) client_negotiation.cc:769] Sending connection context
0924 11:37:04.184595 (+ 393us) client_negotiation.cc:240] Negotiation successful
0924 11:37:04.185297 (+ 702us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":338,"threads_started":1}
I20260924 11:37:04.188463 20241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:04.155815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:33079 (local address 127.6.212.62:35281)
0924 11:37:04.156343 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:04.156349 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:04.157771 (+ 1422us) server_negotiation.cc:407] Connection header received
0924 11:37:04.160428 (+ 2657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:04.160435 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:04.160525 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:04.160628 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:04.162740 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:04.164431 (+ 1691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:04.167090 (+ 2659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:04.167991 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:04.168157 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:04.173021 (+ 4864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:04.173031 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:04.173039 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:04.173087 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:04.180640 (+ 7553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:04.181954 (+ 1314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:04.181961 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:04.181965 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:04.182158 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:04.182815 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:04.182823 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:04.182829 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:04.183289 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:04.183558 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:04.187651 (+ 4093us) server_negotiation.cc:299] Negotiation successful
0924 11:37:04.187980 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":257,"threads_started":1}
I20260924 11:37:04.191110 20239 heartbeater.cc:347] Connected to a master server at 127.6.212.62:35281
I20260924 11:37:04.191504 20239 heartbeater.cc:464] Registering TS with master...
I20260924 11:37:04.192548 20239 heartbeater.cc:511] Master 127.6.212.62:35281 requested a full tablet report, sending...
I20260924 11:37:04.194897 19898 ts_manager.cc:194] Registered new tserver with Master: 14cb8e071a7a4fce8653d37580a207b6 (127.6.212.2:36961)
I20260924 11:37:04.197119 19898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.2@KRBTEST.COM'} at 127.6.212.2:33079
I20260924 11:37:04.491199 20095 heartbeater.cc:464] Registering TS with master...
I20260924 11:37:04.491698 20095 heartbeater.cc:511] Master 127.6.212.62:35281 requested a full tablet report, sending...
I20260924 11:37:04.492996 19898 ts_manager.cc:194] Registered new tserver with Master: f1bdad89e3fc460b98a4cd18266ae2ac (127.6.212.1:35781)
I20260924 11:37:04.494254 19898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.1@KRBTEST.COM'} at 127.6.212.1:44411
W20260924 11:37:04.579210 20243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:37:04.579790 20243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:37:04.579977 20243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:37:04.591638 20243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:37:04.591866 20243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:37:04.592092 20243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.3
I20260924 11:37:04.607715 20243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:34145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.212.20:57751/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab
--principal=kudu/127.6.212.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.3:41513
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.212.3
--webserver_port=32859
--tserver_master_addrs=127.6.212.62:35281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.3
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:37:04.609635 20243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:37:04.611446 20243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Sep 24 11:37:04 dist-test-slave-cq7c krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249824, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for HTTP/127.6.212.20@KRBTEST.COM
W20260924 11:37:04.636189 20249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:37:04.647835 20250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:37:04.648074 20243 server_base.cc:1034] running on GCE node
W20260924 11:37:04.648634 20252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:37:04.649749 20243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:37:04.657235 20243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:37:04.658517 20243 hybrid_clock.cc:648] HybridClock initialized: now 1790249824658451 us; error 63 us; skew 500 ppm
I20260924 11:37:04.658977 20243 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:37:04 dist-test-slave-cq7c krb5kdc[18887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249824, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:37:04.663237 20243 init.cc:377] Logged in from keytab as kudu/127.6.212.3@KRBTEST.COM (short username kudu)
I20260924 11:37:04.686877 20243 webserver.cc:460] Webserver started at http://127.6.212.3:32859/ using document root <none> and password file <none>
I20260924 11:37:04.688043 20243 fs_manager.cc:359] Metadata directory not provided
I20260924 11:37:04.688295 20243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 11:37:04 dist-test-slave-cq7c krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249824, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.3@KRBTEST.COM for HTTP/127.6.212.20@KRBTEST.COM
I20260924 11:37:04.735998 20243 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 11:37:04.746173 20260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:04.748158 20243 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.003s sys 0.007s
I20260924 11:37:04.748414 20243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-2/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-2/wal
uuid: "865b55a6de8743199673497e5298532e"
format_stamp: "Formatted at 2026-09-24 11:37:01 on dist-test-slave-cq7c"
server_key: "c9c6e99658503b96b3b00ff414643849"
server_key_iv: "9e961ec706aca7a7af6d58433dde7977"
server_key_version: "kuduclusterkey@0"
I20260924 11:37:04.749935 20243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:37:04.791997 20243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:37:04.797190 20243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:37:04.797550 20243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:37:04.808785 20243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:37:04.810727 20243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:37:04.810843 20243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:04.810999 20243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:37:04.811085 20243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:04.856564 20243 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.3:41513
I20260924 11:37:04.856650 20372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.3:41513 every 8 connection(s)
I20260924 11:37:04.858495 20243 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
I20260924 11:37:04.864254 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 20243
Sep 24 11:37:04 dist-test-slave-cq7c krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249824, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.3@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:37:04.912604 20375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:04.865608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56168 (local address 127.6.212.62:35281)
0924 11:37:04.891811 (+ 26203us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:04.891818 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:04.891838 (+ 20us) server_negotiation.cc:407] Connection header received
0924 11:37:04.891901 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:04.891906 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:04.891990 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:04.892132 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:04.894331 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:04.895554 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:04.897575 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:04.898194 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:04.898319 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:04.903213 (+ 4894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:04.903220 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:04.903225 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:04.903264 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:04.907902 (+ 4638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:04.909089 (+ 1187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:04.909096 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:04.909102 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:04.909347 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:04.909857 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:04.909863 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:04.909866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:04.910350 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:04.911717 (+ 1367us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:04.911848 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 11:37:04.912198 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26060,"thread_start_us":22147,"threads_started":1}
I20260924 11:37:04.914225 20378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:04.861423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:32963 (local address 127.6.212.62:35281)
0924 11:37:04.887756 (+ 26333us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:04.887763 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:04.888993 (+ 1230us) server_negotiation.cc:407] Connection header received
0924 11:37:04.892556 (+ 3563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:04.892561 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:04.892634 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:04.892735 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:04.894966 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:04.896275 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:04.898931 (+ 2656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:04.899601 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:04.899714 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:04.904185 (+ 4471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:04.904191 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:04.904195 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:04.904245 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:04.908857 (+ 4612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:04.909744 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:04.909748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:04.909752 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:04.909872 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:04.910378 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:04.910383 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:04.910386 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:04.910700 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:04.910813 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:04.913611 (+ 2798us) server_negotiation.cc:299] Negotiation successful
0924 11:37:04.913816 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26190,"thread_start_us":30081,"threads_started":1}
I20260924 11:37:04.914408 20376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:04.862297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35281 (local address 127.6.212.3:32963)
0924 11:37:04.888509 (+ 26212us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:04.888586 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0924 11:37:04.891110 (+ 2524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:37:04.893232 (+ 2122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:37:04.893246 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:37:04.893862 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:37:04.894713 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:04.894740 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:04.896451 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:04.896456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:04.898741 (+ 2285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:04.898758 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:04.900048 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:04.900056 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:04.900336 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:04.901087 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:37:04.901123 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:37:04.903890 (+ 2767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:37:04.909029 (+ 5139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:04.909039 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:04.909057 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:04.909604 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:04.910000 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:04.910004 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:04.910007 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:04.910255 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:04.910888 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:37:04.910895 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:37:04.911323 (+ 428us) client_negotiation.cc:769] Sending connection context
0924 11:37:04.911832 (+ 509us) client_negotiation.cc:240] Negotiation successful
0924 11:37:04.912384 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25839,"thread_start_us":25572,"threads_started":1}
I20260924 11:37:04.916217 20373 heartbeater.cc:347] Connected to a master server at 127.6.212.62:35281
I20260924 11:37:04.916540 20373 heartbeater.cc:464] Registering TS with master...
I20260924 11:37:04.917393 20373 heartbeater.cc:511] Master 127.6.212.62:35281 requested a full tablet report, sending...
I20260924 11:37:04.919337 19897 ts_manager.cc:194] Registered new tserver with Master: 865b55a6de8743199673497e5298532e (127.6.212.3:41513)
I20260924 11:37:04.920668 19897 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.3@KRBTEST.COM'} at 127.6.212.3:32963
I20260924 11:37:04.928447 6992 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:37:04.951347 20378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:04.934303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56170 (local address 127.6.212.62:35281)
0924 11:37:04.934474 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:04.934479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:04.935906 (+ 1427us) server_negotiation.cc:407] Connection header received
0924 11:37:04.936126 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:04.936135 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:04.936235 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:04.936349 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:04.938149 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:04.939194 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:04.941063 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:04.941644 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:04.941740 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:04.945053 (+ 3313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:04.945060 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:04.945065 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:04.945105 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:04.948223 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:04.949182 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:04.949188 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:04.949191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:04.949341 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:04.949783 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:04.949788 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:04.949791 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:04.950149 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:04.950383 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:04.950777 (+ 394us) server_negotiation.cc:299] Negotiation successful
0924 11:37:04.951021 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 11:37:04.959439 19897 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:37:04.970119 19897 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Sep 24 11:37:04 dist-test-slave-cq7c krb5kdc[18887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249824, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for HTTP/127.6.212.20@KRBTEST.COM
I20260924 11:37:05.075839 20389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:05.005832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:35781 (local address 127.0.0.1:42282)
0924 11:37:05.057137 (+ 51305us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:05.057160 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 11:37:05.057309 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:37:05.060431 (+ 3122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:37:05.060447 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:37:05.060475 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:37:05.060870 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:05.060883 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:05.063121 (+ 2238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:05.063128 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:05.069569 (+ 6441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:05.069590 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:05.074834 (+ 5244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:05.074841 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:05.075063 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:05.075131 (+ 68us) client_negotiation.cc:769] Sending connection context
0924 11:37:05.075255 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 11:37:05.075482 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51149,"thread_start_us":47138,"threads_started":1}
I20260924 11:37:05.078553 20393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:05.007189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42282 (local address 127.6.212.1:35781)
0924 11:37:05.051282 (+ 44093us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:05.051294 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:05.057494 (+ 6200us) server_negotiation.cc:407] Connection header received
0924 11:37:05.057746 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:05.057757 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:05.058205 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:05.058434 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:37:05.061126 (+ 2692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:05.062917 (+ 1791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:05.070101 (+ 7184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:05.073295 (+ 3194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:05.073434 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:05.073678 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:05.076301 (+ 2623us) server_negotiation.cc:299] Negotiation successful
0924 11:37:05.078203 (+ 1902us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43852,"thread_start_us":45515,"threads_started":1}
I20260924 11:37:05.081813 20394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:05.005814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.3:41513 (local address 127.0.0.1:38060)
0924 11:37:05.052168 (+ 46354us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:05.052240 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0924 11:37:05.052470 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:37:05.059743 (+ 7273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:37:05.059752 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:37:05.059785 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:37:05.060231 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:05.060248 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:05.065231 (+ 4983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:05.065237 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:05.072383 (+ 7146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:05.072401 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:05.075813 (+ 3412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:05.075821 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:05.081180 (+ 5359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:05.081221 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 11:37:05.081311 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 11:37:05.081475 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45890,"thread_start_us":50737,"threads_started":1}
I20260924 11:37:05.084438 20030 tablet_service.cc:1467] Processing CreateTablet for tablet 37389433c65147b5bc49bf201750f0c8 (DEFAULT_TABLE table=test-table [id=ba9bbde19a124f23baa5fe25fd24ccc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:37:05.086287 20030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37389433c65147b5bc49bf201750f0c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:37:05.089035 20390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:05.006611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38060 (local address 127.6.212.3:41513)
0924 11:37:05.056346 (+ 49735us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:05.056356 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:05.056410 (+ 54us) server_negotiation.cc:407] Connection header received
0924 11:37:05.058882 (+ 2472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:05.058906 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:05.059154 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:05.059375 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:37:05.063318 (+ 3943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:05.065037 (+ 1719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:05.073577 (+ 8540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:05.075186 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:05.075337 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:05.075616 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:05.088351 (+ 12735us) server_negotiation.cc:299] Negotiation successful
0924 11:37:05.088585 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49525,"thread_start_us":63075,"threads_started":1}
I20260924 11:37:05.090677 20391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:05.007039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.2:36961 (local address 127.0.0.1:55564)
0924 11:37:05.072223 (+ 65184us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:05.072250 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 11:37:05.072413 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:37:05.077572 (+ 5159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:37:05.077577 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:37:05.077605 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:37:05.077910 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:05.077922 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:05.085324 (+ 7402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:05.085330 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:05.087398 (+ 2068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:05.087426 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:05.089848 (+ 2422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:05.089857 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:05.090071 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:05.090116 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 11:37:05.090203 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 11:37:05.090349 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65048,"thread_start_us":64272,"threads_started":1}
I20260924 11:37:05.091451 20392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:05.007133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55564 (local address 127.6.212.2:36961)
0924 11:37:05.072007 (+ 64874us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:05.072017 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:05.072566 (+ 549us) server_negotiation.cc:407] Connection header received
0924 11:37:05.072687 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:05.072696 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:05.073034 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:05.073271 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:37:05.082093 (+ 8822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:05.085132 (+ 3039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:05.087625 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:05.089290 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:05.089468 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:05.089685 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:05.090968 (+ 1283us) server_negotiation.cc:299] Negotiation successful
0924 11:37:05.091144 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64675,"thread_start_us":64553,"threads_started":1}
I20260924 11:37:05.095729 20308 tablet_service.cc:1467] Processing CreateTablet for tablet 37389433c65147b5bc49bf201750f0c8 (DEFAULT_TABLE table=test-table [id=ba9bbde19a124f23baa5fe25fd24ccc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:37:05.095682 20174 tablet_service.cc:1467] Processing CreateTablet for tablet 37389433c65147b5bc49bf201750f0c8 (DEFAULT_TABLE table=test-table [id=ba9bbde19a124f23baa5fe25fd24ccc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:37:05.097276 20308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37389433c65147b5bc49bf201750f0c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:37:05.097276 20174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37389433c65147b5bc49bf201750f0c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:37:05.129649 20395 tablet_bootstrap.cc:492] T 37389433c65147b5bc49bf201750f0c8 P f1bdad89e3fc460b98a4cd18266ae2ac: Bootstrap starting.
I20260924 11:37:05.136089 20395 tablet_bootstrap.cc:654] T 37389433c65147b5bc49bf201750f0c8 P f1bdad89e3fc460b98a4cd18266ae2ac: Neither blocks nor log segments found. Creating new log.
I20260924 11:37:05.136740 20397 tablet_bootstrap.cc:492] T 37389433c65147b5bc49bf201750f0c8 P 14cb8e071a7a4fce8653d37580a207b6: Bootstrap starting.
I20260924 11:37:05.137390 20395 log.cc:824] T 37389433c65147b5bc49bf201750f0c8 P f1bdad89e3fc460b98a4cd18266ae2ac: Log is configured to *not* fsync() on all Append() calls
I20260924 11:37:05.138499 20396 tablet_bootstrap.cc:492] T 37389433c65147b5bc49bf201750f0c8 P 865b55a6de8743199673497e5298532e: Bootstrap starting.
I20260924 11:37:05.140944 20397 tablet_bootstrap.cc:654] T 37389433c65147b5bc49bf201750f0c8 P 14cb8e071a7a4fce8653d37580a207b6: Neither blocks nor log segments found. Creating new log.
I20260924 11:37:05.141865 20396 tablet_bootstrap.cc:654] T 37389433c65147b5bc49bf201750f0c8 P 865b55a6de8743199673497e5298532e: Neither blocks nor log segments found. Creating new log.
I20260924 11:37:05.142419 20397 log.cc:824] T 37389433c65147b5bc49bf201750f0c8 P 14cb8e071a7a4fce8653d37580a207b6: Log is configured to *not* fsync() on all Append() calls
I20260924 11:37:05.143118 20396 log.cc:824] T 37389433c65147b5bc49bf201750f0c8 P 865b55a6de8743199673497e5298532e: Log is configured to *not* fsync() on all Append() calls
I20260924 11:37:05.160311 20395 tablet_bootstrap.cc:492] T 37389433c65147b5bc49bf201750f0c8 P f1bdad89e3fc460b98a4cd18266ae2ac: No bootstrap required, opened a new log
I20260924 11:37:05.160763 20395 ts_tablet_manager.cc:1397] T 37389433c65147b5bc49bf201750f0c8 P f1bdad89e3fc460b98a4cd18266ae2ac: Time spent bootstrapping tablet: real 0.031s user 0.006s sys 0.003s
I20260924 11:37:05.184028 20397 tablet_bootstrap.cc:492] T 37389433c65147b5bc49bf201750f0c8 P 14cb8e071a7a4fce8653d37580a207b6: No bootstrap required, opened a new log
I20260924 11:37:05.184468 20397 ts_tablet_manager.cc:1397] T 37389433c65147b5bc49bf201750f0c8 P 14cb8e071a7a4fce8653d37580a207b6: Time spent bootstrapping tablet: real 0.049s user 0.018s sys 0.028s
I20260924 11:37:05.185983 20395 raft_consensus.cc:348] T 37389433c65147b5bc49bf201750f0c8 P f1bdad89e3fc460b98a4cd18266ae2ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865b55a6de8743199673497e5298532e" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41513 } } peers { permanent_uuid: "f1bdad89e3fc460b98a4cd18266ae2ac" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 35781 } } peers { permanent_uuid: "14cb8e071a7a4fce8653d37580a207b6" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36961 } }
I20260924 11:37:05.186599 20395 raft_consensus.cc:374] T 37389433c65147b5bc49bf201750f0c8 P f1bdad89e3fc460b98a4cd18266ae2ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:37:05.186740 20395 raft_consensus.cc:729] T 37389433c65147b5bc49bf201750f0c8 P f1bdad89e3fc460b98a4cd18266ae2ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1bdad89e3fc460b98a4cd18266ae2ac, State: Initialized, Role: FOLLOWER
I20260924 11:37:05.187558 20395 consensus_queue.cc:260] T 37389433c65147b5bc49bf201750f0c8 P f1bdad89e3fc460b98a4cd18266ae2ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865b55a6de8743199673497e5298532e" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41513 } } peers { permanent_uuid: "f1bdad89e3fc460b98a4cd18266ae2ac" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 35781 } } peers { permanent_uuid: "14cb8e071a7a4fce8653d37580a207b6" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36961 } }
I20260924 11:37:05.188354 20396 tablet_bootstrap.cc:492] T 37389433c65147b5bc49bf201750f0c8 P 865b55a6de8743199673497e5298532e: No bootstrap required, opened a new log
I20260924 11:37:05.194590 20396 ts_tablet_manager.cc:1397] T 37389433c65147b5bc49bf201750f0c8 P 865b55a6de8743199673497e5298532e: Time spent bootstrapping tablet: real 0.056s user 0.013s sys 0.016s
I20260924 11:37:05.195811 20395 ts_tablet_manager.cc:1428] T 37389433c65147b5bc49bf201750f0c8 P f1bdad89e3fc460b98a4cd18266ae2ac: Time spent starting tablet: real 0.035s user 0.020s sys 0.004s
I20260924 11:37:05.195654 20397 raft_consensus.cc:348] T 37389433c65147b5bc49bf201750f0c8 P 14cb8e071a7a4fce8653d37580a207b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865b55a6de8743199673497e5298532e" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41513 } } peers { permanent_uuid: "f1bdad89e3fc460b98a4cd18266ae2ac" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 35781 } } peers { permanent_uuid: "14cb8e071a7a4fce8653d37580a207b6" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36961 } }
I20260924 11:37:05.196206 20095 heartbeater.cc:503] Master 127.6.212.62:35281 was elected leader, sending a full tablet report...
I20260924 11:37:05.196208 20397 raft_consensus.cc:374] T 37389433c65147b5bc49bf201750f0c8 P 14cb8e071a7a4fce8653d37580a207b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:37:05.196321 20397 raft_consensus.cc:729] T 37389433c65147b5bc49bf201750f0c8 P 14cb8e071a7a4fce8653d37580a207b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14cb8e071a7a4fce8653d37580a207b6, State: Initialized, Role: FOLLOWER
I20260924 11:37:05.197075 20397 consensus_queue.cc:260] T 37389433c65147b5bc49bf201750f0c8 P 14cb8e071a7a4fce8653d37580a207b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865b55a6de8743199673497e5298532e" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41513 } } peers { permanent_uuid: "f1bdad89e3fc460b98a4cd18266ae2ac" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 35781 } } peers { permanent_uuid: "14cb8e071a7a4fce8653d37580a207b6" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36961 } }
I20260924 11:37:05.202198 20239 heartbeater.cc:503] Master 127.6.212.62:35281 was elected leader, sending a full tablet report...
I20260924 11:37:05.207561 20397 ts_tablet_manager.cc:1428] T 37389433c65147b5bc49bf201750f0c8 P 14cb8e071a7a4fce8653d37580a207b6: Time spent starting tablet: real 0.023s user 0.015s sys 0.007s
I20260924 11:37:05.216058 20396 raft_consensus.cc:348] T 37389433c65147b5bc49bf201750f0c8 P 865b55a6de8743199673497e5298532e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865b55a6de8743199673497e5298532e" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41513 } } peers { permanent_uuid: "f1bdad89e3fc460b98a4cd18266ae2ac" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 35781 } } peers { permanent_uuid: "14cb8e071a7a4fce8653d37580a207b6" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36961 } }
I20260924 11:37:05.216670 20396 raft_consensus.cc:374] T 37389433c65147b5bc49bf201750f0c8 P 865b55a6de8743199673497e5298532e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:37:05.216866 20396 raft_consensus.cc:729] T 37389433c65147b5bc49bf201750f0c8 P 865b55a6de8743199673497e5298532e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 865b55a6de8743199673497e5298532e, State: Initialized, Role: FOLLOWER
I20260924 11:37:05.217736 20396 consensus_queue.cc:260] T 37389433c65147b5bc49bf201750f0c8 P 865b55a6de8743199673497e5298532e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865b55a6de8743199673497e5298532e" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41513 } } peers { permanent_uuid: "f1bdad89e3fc460b98a4cd18266ae2ac" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 35781 } } peers { permanent_uuid: "14cb8e071a7a4fce8653d37580a207b6" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36961 } }
I20260924 11:37:05.218954 20373 heartbeater.cc:503] Master 127.6.212.62:35281 was elected leader, sending a full tablet report...
I20260924 11:37:05.219762 20396 ts_tablet_manager.cc:1428] T 37389433c65147b5bc49bf201750f0c8 P 865b55a6de8743199673497e5298532e: Time spent starting tablet: real 0.025s user 0.008s sys 0.004s
I20260924 11:37:05.222766 20403 raft_consensus.cc:482] T 37389433c65147b5bc49bf201750f0c8 P 865b55a6de8743199673497e5298532e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:37:05.223129 20403 raft_consensus.cc:504] T 37389433c65147b5bc49bf201750f0c8 P 865b55a6de8743199673497e5298532e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865b55a6de8743199673497e5298532e" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41513 } } peers { permanent_uuid: "f1bdad89e3fc460b98a4cd18266ae2ac" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 35781 } } peers { permanent_uuid: "14cb8e071a7a4fce8653d37580a207b6" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36961 } }
I20260924 11:37:05.234319 20376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:05.225857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.2:36961 (local address 127.6.212.3:42237)
0924 11:37:05.226364 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:05.226384 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 11:37:05.226496 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:37:05.227467 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:37:05.227473 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:37:05.227501 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:37:05.227729 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:05.227741 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:05.229193 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:05.229198 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:05.231295 (+ 2097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:05.231310 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:05.233457 (+ 2147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:05.233464 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:05.233694 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:05.233731 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 11:37:05.233831 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 11:37:05.233981 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381}
I20260924 11:37:05.234941 20392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:05.226717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:42237 (local address 127.6.212.2:36961)
0924 11:37:05.226911 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:05.226917 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:05.226937 (+ 20us) server_negotiation.cc:407] Connection header received
0924 11:37:05.227029 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:05.227035 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:05.227120 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:05.227236 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:37:05.227904 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:05.229027 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:05.231609 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:05.232889 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:05.233045 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:05.233157 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:05.234544 (+ 1387us) server_negotiation.cc:299] Negotiation successful
0924 11:37:05.234690 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 11:37:05.236246 20194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37389433c65147b5bc49bf201750f0c8" candidate_uuid: "865b55a6de8743199673497e5298532e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14cb8e071a7a4fce8653d37580a207b6" is_pre_election: true
I20260924 11:37:05.236869 20194 raft_consensus.cc:2454] T 37389433c65147b5bc49bf201750f0c8 P 14cb8e071a7a4fce8653d37580a207b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 865b55a6de8743199673497e5298532e in term 0.
I20260924 11:37:05.237776 20262 leader_election.cc:302] T 37389433c65147b5bc49bf201750f0c8 P 865b55a6de8743199673497e5298532e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14cb8e071a7a4fce8653d37580a207b6, 865b55a6de8743199673497e5298532e; no voters:
I20260924 11:37:05.243345 20403 leader_election.cc:288] T 37389433c65147b5bc49bf201750f0c8 P 865b55a6de8743199673497e5298532e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1bdad89e3fc460b98a4cd18266ae2ac (127.6.212.1:35781), 14cb8e071a7a4fce8653d37580a207b6 (127.6.212.2:36961)
I20260924 11:37:05.243918 20403 raft_consensus.cc:2781] T 37389433c65147b5bc49bf201750f0c8 P 865b55a6de8743199673497e5298532e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:37:05.244441 20403 raft_consensus.cc:482] T 37389433c65147b5bc49bf201750f0c8 P 865b55a6de8743199673497e5298532e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:37:05.244889 20403 raft_consensus.cc:3037] T 37389433c65147b5bc49bf201750f0c8 P 865b55a6de8743199673497e5298532e [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:37:05.245342 20393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:05.226584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:55441 (local address 127.6.212.1:35781)
0924 11:37:05.226769 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:05.226775 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:05.238616 (+ 11841us) server_negotiation.cc:407] Connection header received
0924 11:37:05.239021 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:05.239027 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:05.239106 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:05.239195 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:37:05.239777 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:05.240848 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:05.242964 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:05.244183 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:05.244291 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:05.244395 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:05.244974 (+ 579us) server_negotiation.cc:299] Negotiation successful
0924 11:37:05.245105 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 11:37:05.246096 20376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:05.226089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:35781 (local address 127.6.212.3:55441)
0924 11:37:05.238495 (+ 12406us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:05.238517 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 11:37:05.238885 (+ 368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:37:05.239354 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:37:05.239379 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:37:05.239407 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:37:05.239602 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:05.239613 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:05.241026 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:05.241031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:05.242787 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:05.242801 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:05.244576 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:05.244581 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:05.244785 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:05.244818 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 11:37:05.245596 (+ 778us) client_negotiation.cc:240] Negotiation successful
0924 11:37:05.245741 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12297,"threads_started":1}
I20260924 11:37:05.249452 20403 raft_consensus.cc:504] T 37389433c65147b5bc49bf201750f0c8 P 865b55a6de8743199673497e5298532e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865b55a6de8743199673497e5298532e" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41513 } } peers { permanent_uuid: "f1bdad89e3fc460b98a4cd18266ae2ac" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 35781 } } peers { permanent_uuid: "14cb8e071a7a4fce8653d37580a207b6" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36961 } }
I20260924 11:37:05.252946 20194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37389433c65147b5bc49bf201750f0c8" candidate_uuid: "865b55a6de8743199673497e5298532e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14cb8e071a7a4fce8653d37580a207b6"
I20260924 11:37:05.253244 20194 raft_consensus.cc:3037] T 37389433c65147b5bc49bf201750f0c8 P 14cb8e071a7a4fce8653d37580a207b6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:37:05.255983 20401 raft_consensus.cc:482] T 37389433c65147b5bc49bf201750f0c8 P f1bdad89e3fc460b98a4cd18266ae2ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:37:05.256295 20401 raft_consensus.cc:504] T 37389433c65147b5bc49bf201750f0c8 P f1bdad89e3fc460b98a4cd18266ae2ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865b55a6de8743199673497e5298532e" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41513 } } peers { permanent_uuid: "f1bdad89e3fc460b98a4cd18266ae2ac" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 35781 } } peers { permanent_uuid: "14cb8e071a7a4fce8653d37580a207b6" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36961 } }
I20260924 11:37:05.256618 20194 raft_consensus.cc:2454] T 37389433c65147b5bc49bf201750f0c8 P 14cb8e071a7a4fce8653d37580a207b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 865b55a6de8743199673497e5298532e in term 1.
I20260924 11:37:05.258409 20401 leader_election.cc:288] T 37389433c65147b5bc49bf201750f0c8 P f1bdad89e3fc460b98a4cd18266ae2ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 865b55a6de8743199673497e5298532e (127.6.212.3:41513), 14cb8e071a7a4fce8653d37580a207b6 (127.6.212.2:36961)
I20260924 11:37:05.258445 20262 leader_election.cc:302] T 37389433c65147b5bc49bf201750f0c8 P 865b55a6de8743199673497e5298532e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14cb8e071a7a4fce8653d37580a207b6, 865b55a6de8743199673497e5298532e; no voters:
I20260924 11:37:05.258893 20405 raft_consensus.cc:2781] T 37389433c65147b5bc49bf201750f0c8 P 865b55a6de8743199673497e5298532e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:37:05.259600 20403 leader_election.cc:288] T 37389433c65147b5bc49bf201750f0c8 P 865b55a6de8743199673497e5298532e [CANDIDATE]: Term 1 election: Requested vote from peers f1bdad89e3fc460b98a4cd18266ae2ac (127.6.212.1:35781), 14cb8e071a7a4fce8653d37580a207b6 (127.6.212.2:36961)
I20260924 11:37:05.263554 20049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37389433c65147b5bc49bf201750f0c8" candidate_uuid: "865b55a6de8743199673497e5298532e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1bdad89e3fc460b98a4cd18266ae2ac"
I20260924 11:37:05.263957 20049 raft_consensus.cc:3037] T 37389433c65147b5bc49bf201750f0c8 P f1bdad89e3fc460b98a4cd18266ae2ac [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:37:05.267423 20050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37389433c65147b5bc49bf201750f0c8" candidate_uuid: "865b55a6de8743199673497e5298532e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1bdad89e3fc460b98a4cd18266ae2ac" is_pre_election: true
I20260924 11:37:05.267351 20049 raft_consensus.cc:2454] T 37389433c65147b5bc49bf201750f0c8 P f1bdad89e3fc460b98a4cd18266ae2ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 865b55a6de8743199673497e5298532e in term 1.
I20260924 11:37:05.279668 20405 raft_consensus.cc:686] T 37389433c65147b5bc49bf201750f0c8 P 865b55a6de8743199673497e5298532e [term 1 LEADER]: Becoming Leader. State: Replica: 865b55a6de8743199673497e5298532e, State: Running, Role: LEADER
I20260924 11:37:05.280452 20405 consensus_queue.cc:237] T 37389433c65147b5bc49bf201750f0c8 P 865b55a6de8743199673497e5298532e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865b55a6de8743199673497e5298532e" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41513 } } peers { permanent_uuid: "f1bdad89e3fc460b98a4cd18266ae2ac" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 35781 } } peers { permanent_uuid: "14cb8e071a7a4fce8653d37580a207b6" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36961 } }
I20260924 11:37:05.283967 20392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:05.260091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:48761 (local address 127.6.212.2:36961)
0924 11:37:05.260260 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:05.260267 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:05.261701 (+ 1434us) server_negotiation.cc:407] Connection header received
0924 11:37:05.262610 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:05.262616 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:05.262713 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:05.262777 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:37:05.270423 (+ 7646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:05.271663 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:05.281120 (+ 9457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:05.282293 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:05.282393 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:05.282511 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:05.283378 (+ 867us) server_negotiation.cc:299] Negotiation successful
0924 11:37:05.283598 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 11:37:05.285164 20408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:05.259663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.2:36961 (local address 127.6.212.1:48761)
0924 11:37:05.261603 (+ 1940us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:05.261621 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:37:05.262297 (+ 676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:37:05.269814 (+ 7517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:37:05.269820 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:37:05.269855 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:37:05.270208 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:05.270222 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:05.278451 (+ 8229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:05.278457 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:05.280909 (+ 2452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:05.280924 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:05.282972 (+ 2048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:05.282979 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:05.283132 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:05.283170 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 11:37:05.284467 (+ 1297us) client_negotiation.cc:240] Negotiation successful
0924 11:37:05.284698 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1818,"thread_start_us":1627,"threads_started":1}
I20260924 11:37:05.286242 20194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37389433c65147b5bc49bf201750f0c8" candidate_uuid: "f1bdad89e3fc460b98a4cd18266ae2ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14cb8e071a7a4fce8653d37580a207b6" is_pre_election: true
I20260924 11:37:05.286690 20194 raft_consensus.cc:2379] T 37389433c65147b5bc49bf201750f0c8 P 14cb8e071a7a4fce8653d37580a207b6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f1bdad89e3fc460b98a4cd18266ae2ac in current term 1: Already voted for candidate 865b55a6de8743199673497e5298532e in this term.
I20260924 11:37:05.288725 20390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:05.261574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:43929 (local address 127.6.212.3:41513)
0924 11:37:05.261730 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:05.261735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:05.261753 (+ 18us) server_negotiation.cc:407] Connection header received
0924 11:37:05.261815 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:05.261820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:05.261886 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:05.261973 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:37:05.271573 (+ 9600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:05.272645 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:05.277619 (+ 4974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:05.278855 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:05.278962 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:05.279124 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:05.288129 (+ 9005us) server_negotiation.cc:299] Negotiation successful
0924 11:37:05.288355 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 11:37:05.289707 20407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:05.259072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.3:41513 (local address 127.6.212.1:43929)
0924 11:37:05.260898 (+ 1826us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:05.260918 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 11:37:05.261070 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:37:05.270473 (+ 9403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:37:05.270477 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:37:05.270505 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:37:05.270863 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:05.270876 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:05.272871 (+ 1995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:05.272876 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:05.275018 (+ 2142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:05.275035 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:05.279439 (+ 4404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:05.279448 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:05.282801 (+ 3353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:05.282834 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 11:37:05.289118 (+ 6284us) client_negotiation.cc:240] Negotiation successful
0924 11:37:05.289345 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1685,"thread_start_us":1507,"threads_started":1}
I20260924 11:37:05.290793 20328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37389433c65147b5bc49bf201750f0c8" candidate_uuid: "f1bdad89e3fc460b98a4cd18266ae2ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "865b55a6de8743199673497e5298532e" is_pre_election: true
I20260924 11:37:05.292029 19982 leader_election.cc:302] T 37389433c65147b5bc49bf201750f0c8 P f1bdad89e3fc460b98a4cd18266ae2ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1bdad89e3fc460b98a4cd18266ae2ac; no voters: 14cb8e071a7a4fce8653d37580a207b6, 865b55a6de8743199673497e5298532e
I20260924 11:37:05.292856 20401 raft_consensus.cc:2726] T 37389433c65147b5bc49bf201750f0c8 P f1bdad89e3fc460b98a4cd18266ae2ac [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 11:37:05.294896 19898 catalog_manager.cc:5495] T 37389433c65147b5bc49bf201750f0c8 P 865b55a6de8743199673497e5298532e reported cstate change: term changed from 0 to 1, leader changed from <none> to 865b55a6de8743199673497e5298532e (127.6.212.3). New cstate: current_term: 1 leader_uuid: "865b55a6de8743199673497e5298532e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865b55a6de8743199673497e5298532e" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 41513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1bdad89e3fc460b98a4cd18266ae2ac" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 35781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14cb8e071a7a4fce8653d37580a207b6" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36961 } health_report { overall_health: UNKNOWN } } }
I20260924 11:37:05.367309 20390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:05.360706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38064 (local address 127.6.212.3:41513)
0924 11:37:05.360920 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:05.360926 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:05.360945 (+ 19us) server_negotiation.cc:407] Connection header received
0924 11:37:05.361070 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:05.361076 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:05.361189 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:05.361359 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:37:05.361829 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:05.362750 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:05.364778 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:05.365188 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:05.365285 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:05.366035 (+ 750us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:37:05.366564 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:37:05.366682 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:05.366828 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 11:37:05.366910 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 11:37:05.376931 20049 raft_consensus.cc:1261] T 37389433c65147b5bc49bf201750f0c8 P f1bdad89e3fc460b98a4cd18266ae2ac [term 1 FOLLOWER]: Refusing update from remote peer 865b55a6de8743199673497e5298532e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:37:05.376931 20194 raft_consensus.cc:1261] T 37389433c65147b5bc49bf201750f0c8 P 14cb8e071a7a4fce8653d37580a207b6 [term 1 FOLLOWER]: Refusing update from remote peer 865b55a6de8743199673497e5298532e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:37:05.378156 20406 consensus_queue.cc:1035] T 37389433c65147b5bc49bf201750f0c8 P 865b55a6de8743199673497e5298532e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1bdad89e3fc460b98a4cd18266ae2ac" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 35781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:37:05.378610 20405 consensus_queue.cc:1035] T 37389433c65147b5bc49bf201750f0c8 P 865b55a6de8743199673497e5298532e [LEADER]: Connected to new peer: Peer: permanent_uuid: "14cb8e071a7a4fce8653d37580a207b6" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 11:37:05.381049 20374 tablet.cc:2369] T 37389433c65147b5bc49bf201750f0c8 P 865b55a6de8743199673497e5298532e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:37:05.395256 20418 mvcc.cc:205] Tried to move back new op lower bound from 7332863284733448192 to 7332863284355350528. Current Snapshot: MvccSnapshot[applied={T|T < 7332863284733448192}]
I20260924 11:37:05.411595 20420 mvcc.cc:205] Tried to move back new op lower bound from 7332863284733448192 to 7332863284355350528. Current Snapshot: MvccSnapshot[applied={T|T < 7332863284733448192 or (T in {7332863284733448192})}]
I20260924 11:37:05.426299 19898 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:37:05.426697 19898 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:37:05.431178 19898 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e3c1a78fe3dd4f0f93b601cbdb13361a: Sending DeleteTablet for 3 replicas of tablet 37389433c65147b5bc49bf201750f0c8
I20260924 11:37:05.432798 20030 tablet_service.cc:1514] Processing DeleteTablet for tablet 37389433c65147b5bc49bf201750f0c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:37:05 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:42282
I20260924 11:37:05.432816 20174 tablet_service.cc:1514] Processing DeleteTablet for tablet 37389433c65147b5bc49bf201750f0c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:37:05 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:55564
I20260924 11:37:05.434276 20308 tablet_service.cc:1514] Processing DeleteTablet for tablet 37389433c65147b5bc49bf201750f0c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:37:05 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:38060
I20260924 11:37:05.437909 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 19935
I20260924 11:37:05.447872 20429 tablet_replica.cc:330] T 37389433c65147b5bc49bf201750f0c8 P 865b55a6de8743199673497e5298532e: stopping tablet replica
I20260924 11:37:05.448442 20429 raft_consensus.cc:2229] T 37389433c65147b5bc49bf201750f0c8 P 865b55a6de8743199673497e5298532e [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:37:05.449262 20429 raft_consensus.cc:2258] T 37389433c65147b5bc49bf201750f0c8 P 865b55a6de8743199673497e5298532e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:37:05.451813 20429 ts_tablet_manager.cc:1905] T 37389433c65147b5bc49bf201750f0c8 P 865b55a6de8743199673497e5298532e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:37:05.460259 20429 ts_tablet_manager.cc:1918] T 37389433c65147b5bc49bf201750f0c8 P 865b55a6de8743199673497e5298532e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:37:05.461414 20429 log.cc:1196] T 37389433c65147b5bc49bf201750f0c8 P 865b55a6de8743199673497e5298532e: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-2/wal/wals/37389433c65147b5bc49bf201750f0c8
I20260924 11:37:05.462316 20429 ts_tablet_manager.cc:1939] T 37389433c65147b5bc49bf201750f0c8 P 865b55a6de8743199673497e5298532e: Deleting consensus metadata
I20260924 11:37:05.465791 20428 tablet_replica.cc:330] T 37389433c65147b5bc49bf201750f0c8 P f1bdad89e3fc460b98a4cd18266ae2ac: stopping tablet replica
I20260924 11:37:05.466394 20428 raft_consensus.cc:2229] T 37389433c65147b5bc49bf201750f0c8 P f1bdad89e3fc460b98a4cd18266ae2ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:37:05.468367 20428 raft_consensus.cc:2258] T 37389433c65147b5bc49bf201750f0c8 P f1bdad89e3fc460b98a4cd18266ae2ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:37:05.468984 19882 catalog_manager.cc:4841] TS 865b55a6de8743199673497e5298532e (127.6.212.3:41513): tablet 37389433c65147b5bc49bf201750f0c8 (table test-table [id=ba9bbde19a124f23baa5fe25fd24ccc1]) successfully deleted
I20260924 11:37:05.476382 20427 tablet_replica.cc:330] T 37389433c65147b5bc49bf201750f0c8 P 14cb8e071a7a4fce8653d37580a207b6: stopping tablet replica
I20260924 11:37:05.476989 20427 raft_consensus.cc:2229] T 37389433c65147b5bc49bf201750f0c8 P 14cb8e071a7a4fce8653d37580a207b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:37:05.477588 20427 raft_consensus.cc:2258] T 37389433c65147b5bc49bf201750f0c8 P 14cb8e071a7a4fce8653d37580a207b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:37:05.480624 20427 ts_tablet_manager.cc:1905] T 37389433c65147b5bc49bf201750f0c8 P 14cb8e071a7a4fce8653d37580a207b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:37:05.483145 20393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:05.438711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42290 (local address 127.6.212.1:35781)
0924 11:37:05.438882 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:05.438889 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:05.464624 (+ 25735us) server_negotiation.cc:407] Connection header received
0924 11:37:05.464824 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:05.464830 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:05.464945 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:05.465064 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:05.466794 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:05.467909 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:05.469849 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:05.470436 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:05.470580 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:05.473827 (+ 3247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:05.473841 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:05.473858 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:05.473897 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:05.477568 (+ 3671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:05.478497 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:05.478506 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:05.478522 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:05.478683 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:05.480813 (+ 2130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:05.480819 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:05.480822 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:05.481271 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:05.481587 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:05.482537 (+ 950us) server_negotiation.cc:299] Negotiation successful
0924 11:37:05.482752 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 11:37:05.484176 20090 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:37:05.493021 20427 ts_tablet_manager.cc:1918] T 37389433c65147b5bc49bf201750f0c8 P 14cb8e071a7a4fce8653d37580a207b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:37:05.493194 20427 log.cc:1196] T 37389433c65147b5bc49bf201750f0c8 P 14cb8e071a7a4fce8653d37580a207b6: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790249617057452-6992-0/minicluster-data/ts-1/wal/wals/37389433c65147b5bc49bf201750f0c8
I20260924 11:37:05.493742 20427 ts_tablet_manager.cc:1939] T 37389433c65147b5bc49bf201750f0c8 P 14cb8e071a7a4fce8653d37580a207b6: Deleting consensus metadata
I20260924 11:37:05.495746 19883 catalog_manager.cc:4841] TS 14cb8e071a7a4fce8653d37580a207b6 (127.6.212.2:36961): tablet 37389433c65147b5bc49bf201750f0c8 (table test-table [id=ba9bbde19a124f23baa5fe25fd24ccc1]) successfully deleted
I20260924 11:37:05.646502 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 19935
W20260924 11:37:05.671335 19884 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.212.1:35781 (error 108)
I20260924 11:37:05.678043 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 20099
I20260924 11:37:05.691177 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:05.684318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:35781 (local address 127.0.0.1:42296)
0924 11:37:05.690811 (+ 6493us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:05.690953 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.1:35781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6324,"threads_started":1}
W20260924 11:37:05.692036 19884 catalog_manager.cc:4584] TS f1bdad89e3fc460b98a4cd18266ae2ac (127.6.212.1:35781): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 37389433c65147b5bc49bf201750f0c8: Network error: Client connection negotiation failed: client connection to 127.6.212.1:35781: connect: Connection refused (error 111)
I20260924 11:37:05.697984 20392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:05.678849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55576 (local address 127.6.212.2:36961)
0924 11:37:05.679029 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:05.679035 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:05.679172 (+ 137us) server_negotiation.cc:407] Connection header received
0924 11:37:05.679465 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:05.679475 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:05.679584 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:05.679722 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:05.682211 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:05.683565 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:05.685869 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:05.686781 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:05.686933 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:05.691862 (+ 4929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:05.691883 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:05.691902 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:05.691963 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:05.695034 (+ 3071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:05.695907 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:05.695916 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:05.695919 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:05.696073 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:05.696458 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:05.696464 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:05.696467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:05.696815 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:05.697127 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:05.697400 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 11:37:05.697630 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 11:37:05.698734 20234 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:37:05.745862 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:05.745257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:35781 (local address 127.0.0.1:42300)
0924 11:37:05.745563 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:05.745676 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.1:35781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
I20260924 11:37:05.747457 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:05.746941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:35781 (local address 127.0.0.1:42314)
0924 11:37:05.747193 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:05.747278 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.1:35781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"threads_started":1}
I20260924 11:37:05.802052 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 20099
I20260924 11:37:05.816058 20435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:05.815260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:35781 (local address 127.0.0.1:42316)
0924 11:37:05.815635 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:05.815772 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.1:35781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193}
I20260924 11:37:05.817528 20435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:05.817005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:35781 (local address 127.0.0.1:42330)
0924 11:37:05.817271 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:05.817365 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.1:35781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260924 11:37:05.820525 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 20243
I20260924 11:37:05.835336 20390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:05.821491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38076 (local address 127.6.212.3:41513)
0924 11:37:05.821730 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:05.821735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:05.821749 (+ 14us) server_negotiation.cc:407] Connection header received
0924 11:37:05.821800 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:05.821803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:05.821862 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:05.821994 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:05.823261 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:05.824246 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:05.825508 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:05.825990 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:05.826137 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:05.828823 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:05.828840 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:05.828905 (+ 65us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:05.828956 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:05.832228 (+ 3272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:05.832929 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:05.832940 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:05.832949 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:05.833087 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:05.833477 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:05.833484 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:05.833487 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:05.833983 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:05.834342 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:05.834630 (+ 288us) server_negotiation.cc:299] Negotiation successful
0924 11:37:05.834861 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260924 11:37:05.836193 20368 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:37:05.898420 20435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:05.897843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:35781 (local address 127.0.0.1:42340)
0924 11:37:05.898080 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:05.898197 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.1:35781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 11:37:05.899693 20435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:05.899170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:35781 (local address 127.0.0.1:42356)
0924 11:37:05.899398 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:05.899483 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.1:35781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260924 11:37:05.964702 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 20243
I20260924 11:37:05.980917 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 19865
I20260924 11:37:05.982255 19928 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:37:06.097173 20435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:06.096512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:35781 (local address 127.0.0.1:42360)
0924 11:37:06.096769 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:06.096882 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.1:35781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260924 11:37:06.097441 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 19865
I20260924 11:37:06.122710 6992 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 11:37:06.468868 6992 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 11:37:06.469074 6992 mini_ranger.cc:66] Stopping Ranger...
I20260924 11:37:06.815145 6992 mini_ranger.cc:68] Stopped Ranger
2026-09-24 11:37:06.815 UTC [18917] LOG: received smart shutdown request
2026-09-24 11:37:06.818 UTC [18917] LOG: background worker "logical replication launcher" (PID 18926) exited with exit code 1
2026-09-24 11:37:06.819 UTC [18921] LOG: shutting down
2026-09-24 11:37:07.015 UTC [18917] LOG: database system is shut down
2026-09-24T11:37:07Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (56304 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:37:07 dist-test-slave-cq7c krb5kdc[20446](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:37:07 dist-test-slave-cq7c krb5kdc[20446](info): set up 2 sockets
Sep 24 11:37:07 dist-test-slave-cq7c krb5kdc[20446](info): commencing operation
krb5kdc: starting...
W20260924 11:37:08.263861 6992 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Sep 24 11:37:08 dist-test-slave-cq7c krb5kdc[20446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:37:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:37:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.212.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
I20260924 11:37:08.511706 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:34815
--webserver_interface=127.6.212.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:43017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:34815,127.6.212.61:43399,127.6.212.60:38045
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:37:08.856254 20459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:37:08.856570 20459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:37:08.856747 20459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:37:08.864283 20459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:37:08.864401 20459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:37:08.864508 20459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:37:08.864619 20459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:37:08.864705 20459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:37:08.874894 20459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:43017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:34815,127.6.212.61:43399,127.6.212.60:38045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:34815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.212.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:37:08.876448 20459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:37:08.877856 20459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:37:08.890614 20467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:37:08.890522 20464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:37:08.890578 20465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:37:08.891853 20459 server_base.cc:1034] running on GCE node
I20260924 11:37:08.892803 20459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:37:08.894464 20459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:37:08.895717 20459 hybrid_clock.cc:648] HybridClock initialized: now 1790249828895673 us; error 38 us; skew 500 ppm
I20260924 11:37:08.896107 20459 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:37:08 dist-test-slave-cq7c krb5kdc[20446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249828, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:37:08.905931 20459 init.cc:377] Logged in from keytab as kudu/127.6.212.62@KRBTEST.COM (short username kudu)
I20260924 11:37:08.907816 20459 webserver.cc:460] Webserver started at http://127.6.212.62:38741/ using document root <none> and password file <none>
I20260924 11:37:08.908609 20459 fs_manager.cc:359] Metadata directory not provided
I20260924 11:37:08.908753 20459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:37:08.909117 20459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:37:08.912648 20459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "bcb864052b9743a582a95837c3f503db"
format_stamp: "Formatted at 2026-09-24 11:37:08 on dist-test-slave-cq7c"
server_key: "4d5b9d2488448da8ec76f4959a316565"
server_key_iv: "8699fb0c537ae477e0d208cce1a9fcb9"
server_key_version: "encryptionkey@0"
I20260924 11:37:08.913582 20459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "bcb864052b9743a582a95837c3f503db"
format_stamp: "Formatted at 2026-09-24 11:37:08 on dist-test-slave-cq7c"
server_key: "4d5b9d2488448da8ec76f4959a316565"
server_key_iv: "8699fb0c537ae477e0d208cce1a9fcb9"
server_key_version: "encryptionkey@0"
I20260924 11:37:08.919487 20459 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 11:37:08.923625 20474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:08.925652 20459 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 11:37:08.925850 20459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "bcb864052b9743a582a95837c3f503db"
format_stamp: "Formatted at 2026-09-24 11:37:08 on dist-test-slave-cq7c"
server_key: "4d5b9d2488448da8ec76f4959a316565"
server_key_iv: "8699fb0c537ae477e0d208cce1a9fcb9"
server_key_version: "encryptionkey@0"
I20260924 11:37:08.926098 20459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:37:08.965584 20459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:37:08.969969 20459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:37:08.970294 20459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:37:08.989194 20459 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.62:34815
I20260924 11:37:08.989233 20525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.62:34815 every 8 connection(s)
I20260924 11:37:08.990617 20459 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:37:08.995095 20526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:37:08.996587 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 20459
I20260924 11:37:08.996829 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:37:08.997462 6992 external_mini_cluster.cc:1428] Setting key 6771b70ea26ea782c65cdebfb01b4f4f
I20260924 11:37:09.003777 20526 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.212.62" port: 34815 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.6.212.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.61@KRBTEST.COM" created.
Sep 24 11:37:09 dist-test-slave-cq7c krb5kdc[20446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249828, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:37:09.032054 20530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:09.007446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42850 (local address 127.6.212.62:34815)
0924 11:37:09.009238 (+ 1792us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:09.009247 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:09.009295 (+ 48us) server_negotiation.cc:407] Connection header received
0924 11:37:09.010838 (+ 1543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:09.010844 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:09.011121 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:09.011357 (+ 236us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:09.012798 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:09.014010 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:09.016642 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:09.017291 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:09.017447 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:09.023786 (+ 6339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:09.023795 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:09.023810 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:09.023846 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:09.026792 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:09.027787 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:09.027794 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:09.027796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:09.028000 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:09.028497 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:09.028501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:09.028503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:09.029075 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:09.029289 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:09.030011 (+ 722us) server_negotiation.cc:299] Negotiation successful
0924 11:37:09.030441 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":1560,"spinlock_wait_cycles":47872,"thread_start_us":1312,"threads_started":1}
I20260924 11:37:09.032105 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:09.007445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:34815 (local address 127.0.0.1:42850)
0924 11:37:09.008353 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:09.008400 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 11:37:09.010429 (+ 2029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:37:09.011407 (+ 978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:37:09.011413 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:37:09.012231 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:37:09.012592 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:09.012611 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:09.014229 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:09.014238 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:09.016433 (+ 2195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:09.016472 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:09.017607 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:09.017635 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:09.017859 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:09.018580 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:37:09.018629 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:37:09.023493 (+ 4864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:37:09.026957 (+ 3464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:09.026974 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:09.026992 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:09.027644 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:09.028120 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:09.028128 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:09.028134 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:09.028373 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:09.029249 (+ 876us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:37:09.029261 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:37:09.029544 (+ 283us) client_negotiation.cc:769] Sending connection context
0924 11:37:09.029877 (+ 333us) client_negotiation.cc:240] Negotiation successful
0924 11:37:09.030563 (+ 686us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"mutex_wait_us":207,"thread_start_us":401,"threads_started":1}
I20260924 11:37:09.035326 20526 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.212.61" port: 43399 } has no permanent_uuid. Determining permanent_uuid...
I20260924 11:37:09.036864 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:09.036325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:43399 (local address 127.0.0.1:43694)
0924 11:37:09.036550 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:09.036694 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.61:43399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260924 11:37:09.037223 20476 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.212.61:43399: connect: Connection refused (error 111)
I20260924 11:37:09.039063 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:09.038568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:43399 (local address 127.0.0.1:43696)
0924 11:37:09.038763 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:09.038863 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.61:43399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 11:37:09.039288 20526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.212.61:43399: Network error: Client connection negotiation failed: client connection to 127.6.212.61:43399: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.61.keytab.
Entry for principal kudu/127.6.212.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.61.keytab.
I20260924 11:37:09.053182 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.61:43399
--webserver_interface=127.6.212.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.61.keytab
--principal=kudu/127.6.212.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:43017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:34815,127.6.212.61:43399,127.6.212.60:38045
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:37:09.088711 20526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.212.61" port: 43399 } attempt: 1
I20260924 11:37:09.089968 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:09.089439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:43399 (local address 127.0.0.1:43706)
0924 11:37:09.089678 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:09.089789 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.61:43399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260924 11:37:09.091008 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:09.090618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:43399 (local address 127.0.0.1:43720)
0924 11:37:09.090775 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:09.090857 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.61:43399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 11:37:09.091213 20526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.212.61:43399: Network error: Client connection negotiation failed: client connection to 127.6.212.61:43399: connect: Connection refused (error 111)
I20260924 11:37:09.159502 20526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.212.61" port: 43399 } attempt: 2
I20260924 11:37:09.160871 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:09.160234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:43399 (local address 127.0.0.1:43724)
0924 11:37:09.160511 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:09.160664 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.61:43399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260924 11:37:09.161959 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:09.161573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:43399 (local address 127.0.0.1:43740)
0924 11:37:09.161723 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:09.161817 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.61:43399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 11:37:09.162211 20526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.212.61:43399: Network error: Client connection negotiation failed: client connection to 127.6.212.61:43399: connect: Connection refused (error 111)
I20260924 11:37:09.253520 20526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.212.61" port: 43399 } attempt: 3
I20260924 11:37:09.255014 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:09.254333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:43399 (local address 127.0.0.1:43746)
0924 11:37:09.254655 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:09.254812 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.61:43399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260924 11:37:09.256130 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:09.255719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:43399 (local address 127.0.0.1:43760)
0924 11:37:09.255888 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:09.255967 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.61:43399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260924 11:37:09.256340 20526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.212.61:43399: Network error: Client connection negotiation failed: client connection to 127.6.212.61:43399: connect: Connection refused (error 111)
W20260924 11:37:09.394366 20533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:37:09.394692 20533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:37:09.394783 20533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 11:37:09.399572 20526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.212.61" port: 43399 } attempt: 4
I20260924 11:37:09.401026 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:09.400326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:43399 (local address 127.0.0.1:43770)
0924 11:37:09.400759 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:09.400866 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.61:43399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188}
I20260924 11:37:09.402154 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:09.401765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:43399 (local address 127.0.0.1:43784)
0924 11:37:09.401918 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:09.402010 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.61:43399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 11:37:09.402438 20526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.212.61:43399: Network error: Client connection negotiation failed: client connection to 127.6.212.61:43399: connect: Connection refused (error 111)
W20260924 11:37:09.403154 20533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:37:09.403271 20533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:37:09.403359 20533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:37:09.403481 20533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:37:09.403561 20533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:37:09.413647 20533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:43017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:34815,127.6.212.61:43399,127.6.212.60:38045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.61.keytab
--principal=kudu/127.6.212.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.61:43399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.212.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:37:09.414978 20533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:37:09.416366 20533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:37:09.428463 20538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:37:09.428380 20539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:37:09.428421 20541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:37:09.429586 20533 server_base.cc:1034] running on GCE node
I20260924 11:37:09.430401 20533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:37:09.431849 20533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:37:09.433074 20533 hybrid_clock.cc:648] HybridClock initialized: now 1790249829432990 us; error 50 us; skew 500 ppm
I20260924 11:37:09.433421 20533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:37:09 dist-test-slave-cq7c krb5kdc[20446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249829, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:37:09.436630 20533 init.cc:377] Logged in from keytab as kudu/127.6.212.61@KRBTEST.COM (short username kudu)
I20260924 11:37:09.438011 20533 webserver.cc:460] Webserver started at http://127.6.212.61:44067/ using document root <none> and password file <none>
I20260924 11:37:09.438721 20533 fs_manager.cc:359] Metadata directory not provided
I20260924 11:37:09.438863 20533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:37:09.439178 20533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:37:09.442521 20533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-1/data/instance:
uuid: "b89560fd87714b42876f8bbcd017df34"
format_stamp: "Formatted at 2026-09-24 11:37:09 on dist-test-slave-cq7c"
server_key: "bb167f6afac5a1aaf0d0e5ba12ea5559"
server_key_iv: "cf21b5a625a02392c6589a85d62ffa9c"
server_key_version: "encryptionkey@0"
I20260924 11:37:09.443481 20533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-1/wal/instance:
uuid: "b89560fd87714b42876f8bbcd017df34"
format_stamp: "Formatted at 2026-09-24 11:37:09 on dist-test-slave-cq7c"
server_key: "bb167f6afac5a1aaf0d0e5ba12ea5559"
server_key_iv: "cf21b5a625a02392c6589a85d62ffa9c"
server_key_version: "encryptionkey@0"
I20260924 11:37:09.449693 20533 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 11:37:09.455173 20548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:09.457352 20533 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 11:37:09.457592 20533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-1/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-1/wal
uuid: "b89560fd87714b42876f8bbcd017df34"
format_stamp: "Formatted at 2026-09-24 11:37:09 on dist-test-slave-cq7c"
server_key: "bb167f6afac5a1aaf0d0e5ba12ea5559"
server_key_iv: "cf21b5a625a02392c6589a85d62ffa9c"
server_key_version: "encryptionkey@0"
I20260924 11:37:09.457791 20533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:37:09.477685 20533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:37:09.482563 20533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:37:09.482831 20533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:37:09.503433 20533 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.61:43399
I20260924 11:37:09.503474 20599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.61:43399 every 8 connection(s)
I20260924 11:37:09.505080 20533 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-1/data/info.pb
I20260924 11:37:09.508036 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 20533
I20260924 11:37:09.508270 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-1/wal/instance
I20260924 11:37:09.508724 6992 external_mini_cluster.cc:1428] Setting key 913c5540d0ef8b80dafacf9038c07f73
I20260924 11:37:09.509145 20600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:37:09.515640 20600 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.212.62" port: 34815 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.6.212.60@KRBTEST.COM; defaulting to no policy
Sep 24 11:37:09 dist-test-slave-cq7c krb5kdc[20446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249829, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.61@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.60@KRBTEST.COM" created.
I20260924 11:37:09.538779 20530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:09.518731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42856 (local address 127.6.212.62:34815)
0924 11:37:09.518950 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:09.518956 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:09.520344 (+ 1388us) server_negotiation.cc:407] Connection header received
0924 11:37:09.522286 (+ 1942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:09.522296 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:09.522396 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:09.522529 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:09.524215 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:09.525367 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:09.527206 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:09.528049 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:09.528203 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:09.531977 (+ 3774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:09.531982 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:09.531985 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:09.532094 (+ 109us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:09.535341 (+ 3247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:09.536135 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:09.536142 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:09.536148 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:09.536302 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:09.536733 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:09.536740 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:09.536747 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:09.537157 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:09.537327 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:09.538052 (+ 725us) server_negotiation.cc:299] Negotiation successful
0924 11:37:09.538326 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 11:37:09.539487 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:09.519220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:34815 (local address 127.0.0.1:42856)
0924 11:37:09.520160 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:09.520212 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 11:37:09.521983 (+ 1771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:37:09.522721 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:37:09.522729 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:37:09.523294 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:37:09.524033 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:09.524061 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:09.525555 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:09.525560 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:09.527046 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:09.527058 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:09.528278 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:09.528283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:09.528482 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:09.529203 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:37:09.529226 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:37:09.531725 (+ 2499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:37:09.535539 (+ 3814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:09.535546 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:09.535561 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:09.535990 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:09.536412 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:09.536415 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:09.536417 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:09.536612 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:09.537282 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:37:09.537288 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:37:09.537667 (+ 379us) client_negotiation.cc:769] Sending connection context
0924 11:37:09.537892 (+ 225us) client_negotiation.cc:240] Negotiation successful
0924 11:37:09.538272 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":466,"threads_started":1}
I20260924 11:37:09.541388 20600 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.212.61" port: 43399 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:37:09 dist-test-slave-cq7c krb5kdc[20446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249829, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.61@KRBTEST.COM for kudu/127.6.212.61@KRBTEST.COM
I20260924 11:37:09.556643 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:09.542351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:43399 (local address 127.0.0.1:43786)
0924 11:37:09.542515 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:09.542531 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:37:09.542646 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:37:09.543805 (+ 1159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:37:09.543810 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:37:09.544244 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:37:09.544379 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:09.544386 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:09.545699 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:09.545707 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:09.546626 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:09.546637 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:09.547453 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:09.547461 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:09.547727 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:09.548346 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:37:09.548364 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:37:09.550495 (+ 2131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:37:09.553942 (+ 3447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:09.553948 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:09.553953 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:09.554539 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:09.554950 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:09.554956 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:09.554961 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:09.555174 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:09.555892 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:37:09.555900 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:37:09.556043 (+ 143us) client_negotiation.cc:769] Sending connection context
0924 11:37:09.556160 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 11:37:09.556365 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":5}
I20260924 11:37:09.557021 20605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:09.542653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43786 (local address 127.6.212.61:43399)
0924 11:37:09.543320 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:09.543327 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:09.543351 (+ 24us) server_negotiation.cc:407] Connection header received
0924 11:37:09.543462 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:09.543468 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:09.543665 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:09.543816 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:09.544526 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:09.545564 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:09.546751 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:09.547289 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:09.547450 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:09.550701 (+ 3251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:09.550712 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:09.550725 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:09.550755 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:09.553788 (+ 3033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:09.554688 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:09.554695 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:09.554697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:09.554813 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:09.555290 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:09.555293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:09.555295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:09.555740 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:09.555897 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:09.556332 (+ 435us) server_negotiation.cc:299] Negotiation successful
0924 11:37:09.556536 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":536,"thread_start_us":419,"threads_started":1}
I20260924 11:37:09.559774 20600 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.212.60" port: 38045 } has no permanent_uuid. Determining permanent_uuid...
I20260924 11:37:09.561056 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:09.560474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:38045 (local address 127.0.0.1:55640)
0924 11:37:09.560671 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:09.560795 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:38045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 11:37:09.561429 20550 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.212.60:38045: connect: Connection refused (error 111)
I20260924 11:37:09.563256 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:09.562856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:38045 (local address 127.0.0.1:55646)
0924 11:37:09.563057 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:09.563143 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:38045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 11:37:09.563532 20600 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.212.60:38045: Network error: Client connection negotiation failed: client connection to 127.6.212.60:38045: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.60.keytab.
Entry for principal kudu/127.6.212.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.60.keytab.
I20260924 11:37:09.568521 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.60:38045
--webserver_interface=127.6.212.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.60.keytab
--principal=kudu/127.6.212.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:43017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:34815,127.6.212.61:43399,127.6.212.60:38045
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:37:09.612867 20600 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.212.60" port: 38045 } attempt: 1
I20260924 11:37:09.614179 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:09.613709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:38045 (local address 127.0.0.1:55650)
0924 11:37:09.613911 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:09.614021 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:38045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 11:37:09.615219 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:09.614827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:38045 (local address 127.0.0.1:55652)
0924 11:37:09.615023 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:09.615113 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:38045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 11:37:09.615437 20600 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.212.60:38045: Network error: Client connection negotiation failed: client connection to 127.6.212.60:38045: connect: Connection refused (error 111)
I20260924 11:37:09.683662 20600 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.212.60" port: 38045 } attempt: 2
I20260924 11:37:09.684873 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:09.684411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:38045 (local address 127.0.0.1:55654)
0924 11:37:09.684625 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:09.684718 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:38045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260924 11:37:09.685840 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:09.685502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:38045 (local address 127.0.0.1:55660)
0924 11:37:09.685660 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:09.685732 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:38045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260924 11:37:09.686048 20600 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.212.60:38045: Network error: Client connection negotiation failed: client connection to 127.6.212.60:38045: connect: Connection refused (error 111)
I20260924 11:37:09.701749 20526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.212.61" port: 43399 } attempt: 5
Sep 24 11:37:09 dist-test-slave-cq7c krb5kdc[20446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249828, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for kudu/127.6.212.61@KRBTEST.COM
I20260924 11:37:09.719496 20605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:09.702606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43792 (local address 127.6.212.61:43399)
0924 11:37:09.702801 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:09.702807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:09.702831 (+ 24us) server_negotiation.cc:407] Connection header received
0924 11:37:09.702988 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:09.702993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:09.703076 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:09.703191 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:09.704133 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:09.705222 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:09.707335 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:09.708191 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:09.708347 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:09.713406 (+ 5059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:09.713412 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:09.713415 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:09.713454 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:09.716115 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:09.717535 (+ 1420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:09.717540 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:09.717544 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:09.717670 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:09.718131 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:09.718134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:09.718136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:09.718418 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:09.718529 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:09.719006 (+ 477us) server_negotiation.cc:299] Negotiation successful
0924 11:37:09.719213 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 11:37:09.719569 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:09.702490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:43399 (local address 127.0.0.1:43792)
0924 11:37:09.702712 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:09.702740 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 11:37:09.702846 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:37:09.703249 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:37:09.703253 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:37:09.703743 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:37:09.703937 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:09.703947 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:09.705412 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:09.705420 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:09.707121 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:09.707142 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:09.708426 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:09.708436 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:09.708705 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:09.710212 (+ 1507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:37:09.710242 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:37:09.713150 (+ 2908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:37:09.716326 (+ 3176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:09.716334 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:09.716341 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:09.717060 (+ 719us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:09.717810 (+ 750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:09.717818 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:09.717824 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:09.718016 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:09.718542 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:37:09.718546 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:37:09.718720 (+ 174us) client_negotiation.cc:769] Sending connection context
0924 11:37:09.718897 (+ 177us) client_negotiation.cc:240] Negotiation successful
0924 11:37:09.719185 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260924 11:37:09.720767 20526 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.212.60" port: 38045 } has no permanent_uuid. Determining permanent_uuid...
I20260924 11:37:09.721830 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:09.721373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:38045 (local address 127.0.0.1:55676)
0924 11:37:09.721567 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:09.721663 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:38045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 11:37:09.722769 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:09.722413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:38045 (local address 127.0.0.1:55686)
0924 11:37:09.722555 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:09.722647 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:38045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 11:37:09.722965 20526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.212.60:38045: Network error: Client connection negotiation failed: client connection to 127.6.212.60:38045: connect: Connection refused (error 111)
I20260924 11:37:09.774236 20526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.212.60" port: 38045 } attempt: 1
I20260924 11:37:09.775626 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:09.775001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:38045 (local address 127.0.0.1:55698)
0924 11:37:09.775238 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:09.775409 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:38045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260924 11:37:09.776644 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:09.776292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:38045 (local address 127.0.0.1:55700)
0924 11:37:09.776440 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:09.776525 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:38045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 11:37:09.776839 20526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.212.60:38045: Network error: Client connection negotiation failed: client connection to 127.6.212.60:38045: connect: Connection refused (error 111)
I20260924 11:37:09.777238 20600 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.212.60" port: 38045 } attempt: 3
I20260924 11:37:09.778271 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:09.777778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:38045 (local address 127.0.0.1:55702)
0924 11:37:09.778018 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:09.778138 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:38045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 11:37:09.779397 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:09.778937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:38045 (local address 127.0.0.1:55712)
0924 11:37:09.779173 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:09.779246 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:38045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
W20260924 11:37:09.779584 20600 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.212.60:38045: Network error: Client connection negotiation failed: client connection to 127.6.212.60:38045: connect: Connection refused (error 111)
I20260924 11:37:09.845084 20526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.212.60" port: 38045 } attempt: 2
I20260924 11:37:09.846392 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:09.845774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:38045 (local address 127.0.0.1:55722)
0924 11:37:09.846060 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:09.846186 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:38045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260924 11:37:09.847675 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:09.847234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:38045 (local address 127.0.0.1:55734)
0924 11:37:09.847426 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:09.847534 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:38045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 11:37:09.847890 20526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.212.60:38045: Network error: Client connection negotiation failed: client connection to 127.6.212.60:38045: connect: Connection refused (error 111)
W20260924 11:37:09.909968 20607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:37:09.910264 20607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:37:09.910418 20607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:37:09.917501 20607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:37:09.917610 20607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:37:09.917722 20607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:37:09.917829 20607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:37:09.917912 20607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:37:09.922789 20600 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.212.60" port: 38045 } attempt: 4
I20260924 11:37:09.924088 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:09.923509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:38045 (local address 127.0.0.1:55746)
0924 11:37:09.923785 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:09.923892 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:38045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260924 11:37:09.925207 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:09.924851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:38045 (local address 127.0.0.1:55762)
0924 11:37:09.925001 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:09.925083 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:38045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 11:37:09.925408 20600 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.212.60:38045: Network error: Client connection negotiation failed: client connection to 127.6.212.60:38045: connect: Connection refused (error 111)
I20260924 11:37:09.930109 20607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:43017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:34815,127.6.212.61:43399,127.6.212.60:38045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.60.keytab
--principal=kudu/127.6.212.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.60:38045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.212.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:37:09.931565 20607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:37:09.932955 20607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:37:09.945425 20615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:37:09.945480 20613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:37:09.945477 20612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:37:09.948933 20607 server_base.cc:1034] running on GCE node
I20260924 11:37:09.949787 20607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:37:09.951244 20607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:37:09.952471 20607 hybrid_clock.cc:648] HybridClock initialized: now 1790249829952423 us; error 48 us; skew 500 ppm
I20260924 11:37:09.952828 20607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:37:09.954101 20526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.212.60" port: 38045 } attempt: 3
I20260924 11:37:09.955206 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:09.954670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:38045 (local address 127.0.0.1:55766)
0924 11:37:09.954912 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:09.955013 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:38045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
Sep 24 11:37:09 dist-test-slave-cq7c krb5kdc[20446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249829, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:37:09.956305 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:09.955956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:38045 (local address 127.0.0.1:55780)
0924 11:37:09.956105 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:09.956165 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:38045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 11:37:09.956570 20526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.212.60:38045: Network error: Client connection negotiation failed: client connection to 127.6.212.60:38045: connect: Connection refused (error 111)
I20260924 11:37:09.956812 20607 init.cc:377] Logged in from keytab as kudu/127.6.212.60@KRBTEST.COM (short username kudu)
I20260924 11:37:09.958364 20607 webserver.cc:460] Webserver started at http://127.6.212.60:37791/ using document root <none> and password file <none>
I20260924 11:37:09.959163 20607 fs_manager.cc:359] Metadata directory not provided
I20260924 11:37:09.959299 20607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:37:09.959661 20607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:37:09.962932 20607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-2/data/instance:
uuid: "f83a19c119bb4ae5a896674518cecf65"
format_stamp: "Formatted at 2026-09-24 11:37:09 on dist-test-slave-cq7c"
server_key: "8fa8f6b3c419f56b5d4a714a2faaaedb"
server_key_iv: "227e2dc346891911c19e98e7243dcbb7"
server_key_version: "encryptionkey@0"
I20260924 11:37:09.963778 20607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-2/wal/instance:
uuid: "f83a19c119bb4ae5a896674518cecf65"
format_stamp: "Formatted at 2026-09-24 11:37:09 on dist-test-slave-cq7c"
server_key: "8fa8f6b3c419f56b5d4a714a2faaaedb"
server_key_iv: "227e2dc346891911c19e98e7243dcbb7"
server_key_version: "encryptionkey@0"
I20260924 11:37:09.969144 20607 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 11:37:09.972880 20622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:09.974471 20607 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:37:09.974704 20607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-2/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-2/wal
uuid: "f83a19c119bb4ae5a896674518cecf65"
format_stamp: "Formatted at 2026-09-24 11:37:09 on dist-test-slave-cq7c"
server_key: "8fa8f6b3c419f56b5d4a714a2faaaedb"
server_key_iv: "227e2dc346891911c19e98e7243dcbb7"
server_key_version: "encryptionkey@0"
I20260924 11:37:09.974933 20607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:37:09.988109 20607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:37:09.992568 20607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:37:09.992853 20607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:37:10.012041 20607 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.60:38045
I20260924 11:37:10.012112 20673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.60:38045 every 8 connection(s)
I20260924 11:37:10.013530 20607 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-2/data/info.pb
I20260924 11:37:10.017524 20674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:37:10.022581 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 20607
I20260924 11:37:10.022779 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790249617057452-6992-0/minicluster-data/master-2/wal/instance
I20260924 11:37:10.023427 6992 external_mini_cluster.cc:1428] Setting key a582dc99ee33df4177605b60058084f1
I20260924 11:37:10.024520 20674 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.212.62" port: 34815 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:37:10 dist-test-slave-cq7c krb5kdc[20446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249829, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.60@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
Sep 24 11:37:10 dist-test-slave-cq7c krb5kdc[20446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.60@KRBTEST.COM
I20260924 11:37:10.045179 20530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:10.027582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42868 (local address 127.6.212.62:34815)
0924 11:37:10.027818 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:10.027824 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:10.028680 (+ 856us) server_negotiation.cc:407] Connection header received
0924 11:37:10.029761 (+ 1081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:10.029767 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:10.029849 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:10.030014 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:10.031605 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:10.032410 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:10.034037 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:10.034480 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:10.034614 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:10.038795 (+ 4181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:10.038801 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:10.038804 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:10.038838 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:10.041932 (+ 3094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:10.042750 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:10.042754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:10.042757 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:10.042888 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:10.043331 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:10.043335 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:10.043337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:10.043659 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:10.043832 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:10.044555 (+ 723us) server_negotiation.cc:299] Negotiation successful
0924 11:37:10.044798 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 11:37:10.046370 20678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:10.027505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:34815 (local address 127.0.0.1:42868)
0924 11:37:10.028495 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:10.028552 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 11:37:10.029435 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:37:10.030077 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:37:10.030084 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:37:10.030879 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:37:10.031392 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:10.031426 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:10.032589 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:10.032598 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:10.033894 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:10.033906 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:10.034700 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:10.034706 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:10.034933 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:10.035732 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:37:10.035759 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:37:10.038562 (+ 2803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:37:10.042071 (+ 3509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:10.042085 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:10.042100 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:10.042610 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:10.043006 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:10.043011 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:10.043015 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:10.043205 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:10.043787 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:37:10.043799 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:37:10.044182 (+ 383us) client_negotiation.cc:769] Sending connection context
0924 11:37:10.044461 (+ 279us) client_negotiation.cc:240] Negotiation successful
0924 11:37:10.045027 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"mutex_wait_us":64,"spinlock_wait_cycles":14208,"thread_start_us":515,"threads_started":1}
I20260924 11:37:10.046386 20677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:10.025799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55790 (local address 127.6.212.60:38045)
0924 11:37:10.026696 (+ 897us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:10.026707 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:10.026749 (+ 42us) server_negotiation.cc:407] Connection header received
0924 11:37:10.029435 (+ 2686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:10.029470 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:10.029822 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:10.030100 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:10.032004 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:10.033298 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:10.034896 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:10.035551 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:10.035705 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:10.040134 (+ 4429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:10.040142 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:10.040156 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:10.040188 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:10.042993 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:10.043806 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:10.043812 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:10.043815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:10.043941 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:10.044339 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:10.044343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:10.044345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:10.044696 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:10.045020 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:10.045337 (+ 317us) server_negotiation.cc:299] Negotiation successful
0924 11:37:10.045584 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":436,"threads_started":1}
I20260924 11:37:10.048301 20674 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.212.61" port: 43399 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:37:10 dist-test-slave-cq7c krb5kdc[20446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249829, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.60@KRBTEST.COM for kudu/127.6.212.61@KRBTEST.COM
Sep 24 11:37:10 dist-test-slave-cq7c krb5kdc[20446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:37:10.068608 20678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:10.049143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:43399 (local address 127.0.0.1:43794)
0924 11:37:10.049335 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:10.049356 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 11:37:10.049487 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:37:10.050272 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:37:10.050276 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:37:10.050742 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:37:10.050973 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:10.050987 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:10.052113 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:10.052119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:10.053352 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:10.053368 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:10.054702 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:10.054708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:10.054948 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:10.059330 (+ 4382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:37:10.059413 (+ 83us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:37:10.061809 (+ 2396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:37:10.066038 (+ 4229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:10.066044 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:10.066048 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:10.066487 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:10.066935 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:10.066939 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:10.066942 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:10.067145 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:10.067743 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:37:10.067747 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:37:10.067875 (+ 128us) client_negotiation.cc:769] Sending connection context
0924 11:37:10.068050 (+ 175us) client_negotiation.cc:240] Negotiation successful
0924 11:37:10.068362 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 11:37:10.068737 20605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:10.049823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43794 (local address 127.6.212.61:43399)
0924 11:37:10.049989 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:10.049994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:10.050009 (+ 15us) server_negotiation.cc:407] Connection header received
0924 11:37:10.050059 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:10.050063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:10.050120 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:10.050237 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:10.051181 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:10.051952 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:10.053511 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:10.054353 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:10.054494 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:10.062079 (+ 7585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:10.062085 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:10.062088 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:10.062127 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:10.065872 (+ 3745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:10.066620 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:10.066627 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:10.066632 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:10.066816 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:10.067260 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:10.067263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:10.067266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:10.067622 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:10.067781 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:10.068135 (+ 354us) server_negotiation.cc:299] Negotiation successful
0924 11:37:10.068352 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 11:37:10.069833 20674 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.212.60" port: 38045 } has no permanent_uuid. Determining permanent_uuid...
I20260924 11:37:10.070086 20530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:10.050318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42874 (local address 127.6.212.62:34815)
0924 11:37:10.050481 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:10.050485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:10.050655 (+ 170us) server_negotiation.cc:407] Connection header received
0924 11:37:10.050824 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:10.050827 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:10.050880 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:10.050957 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:10.052388 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:10.053150 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:10.054456 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:10.055091 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:10.055236 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:10.063255 (+ 8019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:10.063265 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:10.063270 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:10.063322 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:10.066715 (+ 3393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:10.067666 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:10.067674 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:10.067678 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:10.067835 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:10.068319 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:10.068324 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:10.068327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:10.068691 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:10.068971 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:10.069501 (+ 530us) server_negotiation.cc:299] Negotiation successful
0924 11:37:10.069729 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Sep 24 11:37:10 dist-test-slave-cq7c krb5kdc[20446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249829, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.60@KRBTEST.COM for kudu/127.6.212.60@KRBTEST.COM
Sep 24 11:37:10 dist-test-slave-cq7c krb5kdc[20446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.61@KRBTEST.COM
I20260924 11:37:10.085827 20677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:10.070684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55800 (local address 127.6.212.60:38045)
0924 11:37:10.070848 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:10.070852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:10.070925 (+ 73us) server_negotiation.cc:407] Connection header received
0924 11:37:10.071115 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:10.071121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:10.071190 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:10.071283 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:10.072201 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:10.073298 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:10.074828 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:10.075458 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:10.075599 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:10.079322 (+ 3723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:10.079329 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:10.079335 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:10.079415 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:10.082817 (+ 3402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:10.083734 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:10.083737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:10.083740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:10.083847 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:10.084247 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:10.084254 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:10.084257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:10.084671 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:10.084851 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:10.085220 (+ 369us) server_negotiation.cc:299] Negotiation successful
0924 11:37:10.085596 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":147,"server-negotiator.queue_time_us":54}
I20260924 11:37:10.085855 20678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:10.070583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:38045 (local address 127.0.0.1:55800)
0924 11:37:10.070854 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:10.070873 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:37:10.070988 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:37:10.071323 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:37:10.071326 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:37:10.071860 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:37:10.072045 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:10.072058 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:10.073457 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:10.073461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:10.074652 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:10.074667 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:10.075604 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:10.075609 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:10.075768 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:10.076432 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:37:10.076450 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:37:10.079102 (+ 2652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:37:10.082977 (+ 3875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:10.082982 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:10.082987 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:10.083609 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:10.083967 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:10.083973 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:10.083976 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:10.084112 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:10.084787 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:37:10.084790 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:37:10.084933 (+ 143us) client_negotiation.cc:769] Sending connection context
0924 11:37:10.085081 (+ 148us) client_negotiation.cc:240] Negotiation successful
0924 11:37:10.085426 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163}
I20260924 11:37:10.089711 20605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:10.071923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43804 (local address 127.6.212.61:43399)
0924 11:37:10.072111 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:10.072118 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:10.072139 (+ 21us) server_negotiation.cc:407] Connection header received
0924 11:37:10.072255 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:10.072259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:10.072315 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:10.072387 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:10.073916 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:10.074972 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:10.076777 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:10.077439 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:10.077602 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:10.083151 (+ 5549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:10.083160 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:10.083169 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:10.083215 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:10.086947 (+ 3732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:10.087751 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:10.087756 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:10.087758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:10.087896 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:10.088288 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:10.088292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:10.088294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:10.088590 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:10.088863 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:10.089138 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 11:37:10.089368 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 11:37:10.091578 20674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f83a19c119bb4ae5a896674518cecf65: Bootstrap starting.
I20260924 11:37:10.094815 6992 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:37:10.095676 20674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f83a19c119bb4ae5a896674518cecf65: Neither blocks nor log segments found. Creating new log.
I20260924 11:37:10.096745 20674 log.cc:824] T 00000000000000000000000000000000 P f83a19c119bb4ae5a896674518cecf65: Log is configured to *not* fsync() on all Append() calls
I20260924 11:37:10.100224 20674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f83a19c119bb4ae5a896674518cecf65: No bootstrap required, opened a new log
I20260924 11:37:10.114135 20674 raft_consensus.cc:348] T 00000000000000000000000000000000 P f83a19c119bb4ae5a896674518cecf65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcb864052b9743a582a95837c3f503db" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 34815 } } peers { permanent_uuid: "b89560fd87714b42876f8bbcd017df34" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 43399 } } peers { permanent_uuid: "f83a19c119bb4ae5a896674518cecf65" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 38045 } }
I20260924 11:37:10.114640 20674 raft_consensus.cc:374] T 00000000000000000000000000000000 P f83a19c119bb4ae5a896674518cecf65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:37:10.114763 20674 raft_consensus.cc:729] T 00000000000000000000000000000000 P f83a19c119bb4ae5a896674518cecf65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83a19c119bb4ae5a896674518cecf65, State: Initialized, Role: FOLLOWER
I20260924 11:37:10.115556 20674 consensus_queue.cc:260] T 00000000000000000000000000000000 P f83a19c119bb4ae5a896674518cecf65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcb864052b9743a582a95837c3f503db" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 34815 } } peers { permanent_uuid: "b89560fd87714b42876f8bbcd017df34" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 43399 } } peers { permanent_uuid: "f83a19c119bb4ae5a896674518cecf65" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 38045 } }
I20260924 11:37:10.116547 20687 sys_catalog.cc:455] T 00000000000000000000000000000000 P f83a19c119bb4ae5a896674518cecf65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcb864052b9743a582a95837c3f503db" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 34815 } } peers { permanent_uuid: "b89560fd87714b42876f8bbcd017df34" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 43399 } } peers { permanent_uuid: "f83a19c119bb4ae5a896674518cecf65" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 38045 } } }
I20260924 11:37:10.117010 20687 sys_catalog.cc:458] T 00000000000000000000000000000000 P f83a19c119bb4ae5a896674518cecf65 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:37:10.118973 20674 sys_catalog.cc:564] T 00000000000000000000000000000000 P f83a19c119bb4ae5a896674518cecf65 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:37:10.127686 20530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:10.099942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42884 (local address 127.6.212.62:34815)
0924 11:37:10.100181 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:10.100187 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:10.107736 (+ 7549us) server_negotiation.cc:407] Connection header received
0924 11:37:10.108276 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:10.108283 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:10.108372 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:10.108443 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:10.110911 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:10.112109 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:10.114347 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:10.115038 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:10.115173 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:10.118772 (+ 3599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:10.118780 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:10.118784 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:10.118844 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:10.123472 (+ 4628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:10.124431 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:10.124440 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:10.124444 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:10.124606 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:10.125300 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:10.125308 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:10.125312 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:10.125709 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:10.126869 (+ 1160us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:10.127007 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 11:37:10.127275 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 11:37:10.128648 20605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:10.099914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43816 (local address 127.6.212.61:43399)
0924 11:37:10.100112 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:10.100119 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:10.100747 (+ 628us) server_negotiation.cc:407] Connection header received
0924 11:37:10.101038 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:10.101049 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:10.101162 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:10.101259 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:10.104558 (+ 3299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:10.106082 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:10.109161 (+ 3079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:10.110080 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:10.110210 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:10.120691 (+ 10481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:10.120701 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:10.120708 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:10.120764 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:10.124719 (+ 3955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:10.125720 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:10.125725 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:10.125728 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:10.125885 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:10.126389 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:10.126393 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:10.126396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:10.126789 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:10.127928 (+ 1139us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:10.128059 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 11:37:10.128306 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 11:37:10.131945 20677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:10.101143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55806 (local address 127.6.212.60:38045)
0924 11:37:10.101358 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:10.101364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:10.111898 (+ 10534us) server_negotiation.cc:407] Connection header received
0924 11:37:10.112055 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:10.112063 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:10.112143 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:10.112267 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:10.114053 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:10.115154 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:10.117379 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:10.118089 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:10.118222 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:10.122247 (+ 4025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:10.122257 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:10.122265 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:10.122321 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:10.127146 (+ 4825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:10.128476 (+ 1330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:10.128482 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:10.128485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:10.128651 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:10.129329 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:10.129334 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:10.129337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:10.129827 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:10.130221 (+ 394us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:10.130861 (+ 640us) server_negotiation.cc:299] Negotiation successful
0924 11:37:10.131408 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 11:37:10.133790 20526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.212.60" port: 38045 } attempt: 4
W20260924 11:37:10.144521 20698 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f83a19c119bb4ae5a896674518cecf65: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 11:37:10.144798 20698 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 24 11:37:10 dist-test-slave-cq7c krb5kdc[20446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249828, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for kudu/127.6.212.60@KRBTEST.COM
I20260924 11:37:10.135558 20607 master_runner.cc:186] Error getting master registration for 127.6.212.62:34815: OK, instance_id { permanent_uuid: "bcb864052b9743a582a95837c3f503db" instance_seqno: 1790249828983568 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 11:37:10.148346 20607 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 11:37:10.153751 20677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:10.134542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55812 (local address 127.6.212.60:38045)
0924 11:37:10.134703 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:10.134708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:10.134727 (+ 19us) server_negotiation.cc:407] Connection header received
0924 11:37:10.134822 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:10.134827 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:10.134911 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:10.135026 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:10.136004 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:10.137621 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:10.141319 (+ 3698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:10.142363 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:10.142497 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:10.146527 (+ 4030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:10.146541 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:10.146545 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:10.146585 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:10.150805 (+ 4220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:10.151651 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:10.151662 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:10.151666 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:10.151791 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:10.152205 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:10.152210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:10.152215 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:10.152546 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:10.152715 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:10.153122 (+ 407us) server_negotiation.cc:299] Negotiation successful
0924 11:37:10.153363 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 11:37:10.153771 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:10.134402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:38045 (local address 127.0.0.1:55812)
0924 11:37:10.134587 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:10.134607 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 11:37:10.134721 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:37:10.135042 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:37:10.135046 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:37:10.135634 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:37:10.135838 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:10.135851 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:10.137869 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:10.137875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:10.139423 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:10.139442 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:10.142754 (+ 3312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:10.142759 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:10.142908 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:10.143599 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:37:10.143618 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:37:10.146292 (+ 2674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:37:10.150984 (+ 4692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:10.150991 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:10.150995 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:10.151503 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:10.151915 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:10.151920 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:10.151925 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:10.152083 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:10.152689 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:37:10.152695 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:37:10.152872 (+ 177us) client_negotiation.cc:769] Sending connection context
0924 11:37:10.153017 (+ 145us) client_negotiation.cc:240] Negotiation successful
0924 11:37:10.153363 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 11:37:10.159137 20526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcb864052b9743a582a95837c3f503db: Bootstrap starting.
I20260924 11:37:10.162675 20526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bcb864052b9743a582a95837c3f503db: Neither blocks nor log segments found. Creating new log.
I20260924 11:37:10.164109 20526 log.cc:824] T 00000000000000000000000000000000 P bcb864052b9743a582a95837c3f503db: Log is configured to *not* fsync() on all Append() calls
I20260924 11:37:10.167337 20526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcb864052b9743a582a95837c3f503db: No bootstrap required, opened a new log
I20260924 11:37:10.175997 20526 raft_consensus.cc:348] T 00000000000000000000000000000000 P bcb864052b9743a582a95837c3f503db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcb864052b9743a582a95837c3f503db" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 34815 } } peers { permanent_uuid: "b89560fd87714b42876f8bbcd017df34" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 43399 } } peers { permanent_uuid: "f83a19c119bb4ae5a896674518cecf65" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 38045 } }
I20260924 11:37:10.176339 20526 raft_consensus.cc:374] T 00000000000000000000000000000000 P bcb864052b9743a582a95837c3f503db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:37:10.176477 20526 raft_consensus.cc:729] T 00000000000000000000000000000000 P bcb864052b9743a582a95837c3f503db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcb864052b9743a582a95837c3f503db, State: Initialized, Role: FOLLOWER
I20260924 11:37:10.177158 20526 consensus_queue.cc:260] T 00000000000000000000000000000000 P bcb864052b9743a582a95837c3f503db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcb864052b9743a582a95837c3f503db" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 34815 } } peers { permanent_uuid: "b89560fd87714b42876f8bbcd017df34" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 43399 } } peers { permanent_uuid: "f83a19c119bb4ae5a896674518cecf65" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 38045 } }
I20260924 11:37:10.178128 20701 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcb864052b9743a582a95837c3f503db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcb864052b9743a582a95837c3f503db" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 34815 } } peers { permanent_uuid: "b89560fd87714b42876f8bbcd017df34" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 43399 } } peers { permanent_uuid: "f83a19c119bb4ae5a896674518cecf65" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 38045 } } }
I20260924 11:37:10.178591 20701 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcb864052b9743a582a95837c3f503db [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:37:10.179303 20526 sys_catalog.cc:564] T 00000000000000000000000000000000 P bcb864052b9743a582a95837c3f503db [sys.catalog]: configured and running, proceeding with master startup.
W20260924 11:37:10.191517 20712 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bcb864052b9743a582a95837c3f503db: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 11:37:10.191702 20712 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 11:37:10.190500 20459 master_runner.cc:186] Error getting master registration for 127.6.212.61:43399: OK, instance_id { permanent_uuid: "b89560fd87714b42876f8bbcd017df34" instance_seqno: 1790249829497499 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 11:37:10.199623 20459 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 11:37:10.223390 20480 tablet.cc:2369] T 00000000000000000000000000000000 P bcb864052b9743a582a95837c3f503db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:37:10.224627 20600 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.212.60" port: 38045 } attempt: 5
Sep 24 11:37:10 dist-test-slave-cq7c krb5kdc[20446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249829, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.61@KRBTEST.COM for kudu/127.6.212.60@KRBTEST.COM
I20260924 11:37:10.244004 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:10.225280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:38045 (local address 127.0.0.1:55818)
0924 11:37:10.225564 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:10.225586 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 11:37:10.225702 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:37:10.226061 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:37:10.226065 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:37:10.226635 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:37:10.226850 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:10.226863 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:10.228436 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:10.228442 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:10.230009 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:10.230028 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:10.231173 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:10.231180 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:10.231453 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:10.232231 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:37:10.232256 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:37:10.235240 (+ 2984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:37:10.241090 (+ 5850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:10.241096 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:10.241100 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:10.241671 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:10.242086 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:10.242092 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:10.242096 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:10.242269 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:10.242903 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:37:10.242909 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:37:10.243078 (+ 169us) client_negotiation.cc:769] Sending connection context
0924 11:37:10.243234 (+ 156us) client_negotiation.cc:240] Negotiation successful
0924 11:37:10.243583 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143}
I20260924 11:37:10.244031 20677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:10.225509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55818 (local address 127.6.212.60:38045)
0924 11:37:10.225697 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:10.225704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:10.225725 (+ 21us) server_negotiation.cc:407] Connection header received
0924 11:37:10.225822 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:10.225827 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:10.225909 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:10.226014 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:10.227031 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:10.228257 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:10.230168 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:10.230994 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:10.231243 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:10.235537 (+ 4294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:10.235544 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:10.235551 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:10.235602 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:10.240930 (+ 5328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:10.241803 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:10.241808 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:10.241812 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:10.241958 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:10.242387 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:10.242391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:10.242395 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:10.242779 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:10.242935 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:10.243356 (+ 421us) server_negotiation.cc:299] Negotiation successful
0924 11:37:10.243654 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260924 11:37:10.245059 20628 tablet.cc:2369] T 00000000000000000000000000000000 P f83a19c119bb4ae5a896674518cecf65: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:37:10.253696 20600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b89560fd87714b42876f8bbcd017df34: Bootstrap starting.
I20260924 11:37:10.258085 20600 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b89560fd87714b42876f8bbcd017df34: Neither blocks nor log segments found. Creating new log.
I20260924 11:37:10.259506 20600 log.cc:824] T 00000000000000000000000000000000 P b89560fd87714b42876f8bbcd017df34: Log is configured to *not* fsync() on all Append() calls
I20260924 11:37:10.263075 20600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b89560fd87714b42876f8bbcd017df34: No bootstrap required, opened a new log
I20260924 11:37:10.276294 20600 raft_consensus.cc:348] T 00000000000000000000000000000000 P b89560fd87714b42876f8bbcd017df34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcb864052b9743a582a95837c3f503db" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 34815 } } peers { permanent_uuid: "b89560fd87714b42876f8bbcd017df34" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 43399 } } peers { permanent_uuid: "f83a19c119bb4ae5a896674518cecf65" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 38045 } }
I20260924 11:37:10.276789 20600 raft_consensus.cc:374] T 00000000000000000000000000000000 P b89560fd87714b42876f8bbcd017df34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:37:10.276911 20600 raft_consensus.cc:729] T 00000000000000000000000000000000 P b89560fd87714b42876f8bbcd017df34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89560fd87714b42876f8bbcd017df34, State: Initialized, Role: FOLLOWER
I20260924 11:37:10.277639 20600 consensus_queue.cc:260] T 00000000000000000000000000000000 P b89560fd87714b42876f8bbcd017df34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcb864052b9743a582a95837c3f503db" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 34815 } } peers { permanent_uuid: "b89560fd87714b42876f8bbcd017df34" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 43399 } } peers { permanent_uuid: "f83a19c119bb4ae5a896674518cecf65" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 38045 } }
I20260924 11:37:10.278748 20715 sys_catalog.cc:455] T 00000000000000000000000000000000 P b89560fd87714b42876f8bbcd017df34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcb864052b9743a582a95837c3f503db" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 34815 } } peers { permanent_uuid: "b89560fd87714b42876f8bbcd017df34" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 43399 } } peers { permanent_uuid: "f83a19c119bb4ae5a896674518cecf65" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 38045 } } }
I20260924 11:37:10.279409 20715 sys_catalog.cc:458] T 00000000000000000000000000000000 P b89560fd87714b42876f8bbcd017df34 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:37:10.280110 20600 sys_catalog.cc:564] T 00000000000000000000000000000000 P b89560fd87714b42876f8bbcd017df34 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:37:10.308418 20533 master_runner.cc:263] No leader master found from master b89560fd87714b42876f8bbcd017df34
I20260924 11:37:10.310672 20533 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 11:37:10.311758 20726 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b89560fd87714b42876f8bbcd017df34: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 11:37:10.311961 20726 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 11:37:10.353030 20701 raft_consensus.cc:482] T 00000000000000000000000000000000 P bcb864052b9743a582a95837c3f503db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:37:10.353404 20701 raft_consensus.cc:504] T 00000000000000000000000000000000 P bcb864052b9743a582a95837c3f503db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcb864052b9743a582a95837c3f503db" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 34815 } } peers { permanent_uuid: "b89560fd87714b42876f8bbcd017df34" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 43399 } } peers { permanent_uuid: "f83a19c119bb4ae5a896674518cecf65" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 38045 } }
I20260924 11:37:10.355166 20701 leader_election.cc:288] T 00000000000000000000000000000000 P bcb864052b9743a582a95837c3f503db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b89560fd87714b42876f8bbcd017df34 (127.6.212.61:43399), f83a19c119bb4ae5a896674518cecf65 (127.6.212.60:38045)
I20260924 11:37:10.356459 20575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bcb864052b9743a582a95837c3f503db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89560fd87714b42876f8bbcd017df34" is_pre_election: true
I20260924 11:37:10.357143 20575 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b89560fd87714b42876f8bbcd017df34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcb864052b9743a582a95837c3f503db in term 0.
I20260924 11:37:10.357892 20476 leader_election.cc:302] T 00000000000000000000000000000000 P bcb864052b9743a582a95837c3f503db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b89560fd87714b42876f8bbcd017df34, bcb864052b9743a582a95837c3f503db; no voters:
I20260924 11:37:10.358397 20701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bcb864052b9743a582a95837c3f503db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:37:10.358588 20701 raft_consensus.cc:482] T 00000000000000000000000000000000 P bcb864052b9743a582a95837c3f503db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:37:10.358714 20701 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bcb864052b9743a582a95837c3f503db [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:37:10.362304 20701 raft_consensus.cc:504] T 00000000000000000000000000000000 P bcb864052b9743a582a95837c3f503db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcb864052b9743a582a95837c3f503db" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 34815 } } peers { permanent_uuid: "b89560fd87714b42876f8bbcd017df34" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 43399 } } peers { permanent_uuid: "f83a19c119bb4ae5a896674518cecf65" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 38045 } }
I20260924 11:37:10.363662 20649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bcb864052b9743a582a95837c3f503db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83a19c119bb4ae5a896674518cecf65" is_pre_election: true
I20260924 11:37:10.364305 20649 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f83a19c119bb4ae5a896674518cecf65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcb864052b9743a582a95837c3f503db in term 0.
I20260924 11:37:10.365228 20575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bcb864052b9743a582a95837c3f503db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89560fd87714b42876f8bbcd017df34"
I20260924 11:37:10.365489 20575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b89560fd87714b42876f8bbcd017df34 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:37:10.377674 20575 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b89560fd87714b42876f8bbcd017df34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcb864052b9743a582a95837c3f503db in term 1.
I20260924 11:37:10.383857 20476 leader_election.cc:302] T 00000000000000000000000000000000 P bcb864052b9743a582a95837c3f503db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b89560fd87714b42876f8bbcd017df34, bcb864052b9743a582a95837c3f503db; no voters:
I20260924 11:37:10.387208 20701 leader_election.cc:288] T 00000000000000000000000000000000 P bcb864052b9743a582a95837c3f503db [CANDIDATE]: Term 1 election: Requested vote from peers b89560fd87714b42876f8bbcd017df34 (127.6.212.61:43399), f83a19c119bb4ae5a896674518cecf65 (127.6.212.60:38045)
I20260924 11:37:10.387511 20701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bcb864052b9743a582a95837c3f503db [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:37:10.387683 20701 raft_consensus.cc:686] T 00000000000000000000000000000000 P bcb864052b9743a582a95837c3f503db [term 1 LEADER]: Becoming Leader. State: Replica: bcb864052b9743a582a95837c3f503db, State: Running, Role: LEADER
I20260924 11:37:10.388298 20701 consensus_queue.cc:237] T 00000000000000000000000000000000 P bcb864052b9743a582a95837c3f503db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcb864052b9743a582a95837c3f503db" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 34815 } } peers { permanent_uuid: "b89560fd87714b42876f8bbcd017df34" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 43399 } } peers { permanent_uuid: "f83a19c119bb4ae5a896674518cecf65" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 38045 } }
I20260924 11:37:10.393860 20649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bcb864052b9743a582a95837c3f503db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f83a19c119bb4ae5a896674518cecf65"
I20260924 11:37:10.394131 20649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f83a19c119bb4ae5a896674518cecf65 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:37:10.398139 20649 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f83a19c119bb4ae5a896674518cecf65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcb864052b9743a582a95837c3f503db in term 1.
I20260924 11:37:10.416812 20728 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcb864052b9743a582a95837c3f503db [sys.catalog]: SysCatalogTable state changed. Reason: New leader bcb864052b9743a582a95837c3f503db. Latest consensus state: current_term: 1 leader_uuid: "bcb864052b9743a582a95837c3f503db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcb864052b9743a582a95837c3f503db" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 34815 } } peers { permanent_uuid: "b89560fd87714b42876f8bbcd017df34" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 43399 } } peers { permanent_uuid: "f83a19c119bb4ae5a896674518cecf65" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 38045 } } }
I20260924 11:37:10.417316 20728 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcb864052b9743a582a95837c3f503db [sys.catalog]: This master's current role is: LEADER
I20260924 11:37:10.427387 20730 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:37:10.429894 20730 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:37:10.452764 20649 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f83a19c119bb4ae5a896674518cecf65 [term 1 FOLLOWER]: Refusing update from remote peer bcb864052b9743a582a95837c3f503db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:37:10.453953 20728 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bcb864052b9743a582a95837c3f503db [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83a19c119bb4ae5a896674518cecf65" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 38045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:37:10.462054 20575 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b89560fd87714b42876f8bbcd017df34 [term 1 FOLLOWER]: Refusing update from remote peer bcb864052b9743a582a95837c3f503db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:37:10.463112 20728 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bcb864052b9743a582a95837c3f503db [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89560fd87714b42876f8bbcd017df34" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 11:37:10.485737 20554 tablet.cc:2369] T 00000000000000000000000000000000 P b89560fd87714b42876f8bbcd017df34: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:37:10.497145 20715 sys_catalog.cc:455] T 00000000000000000000000000000000 P b89560fd87714b42876f8bbcd017df34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bcb864052b9743a582a95837c3f503db. Latest consensus state: current_term: 1 leader_uuid: "bcb864052b9743a582a95837c3f503db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcb864052b9743a582a95837c3f503db" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 34815 } } peers { permanent_uuid: "b89560fd87714b42876f8bbcd017df34" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 43399 } } peers { permanent_uuid: "f83a19c119bb4ae5a896674518cecf65" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 38045 } } }
I20260924 11:37:10.497507 20715 sys_catalog.cc:458] T 00000000000000000000000000000000 P b89560fd87714b42876f8bbcd017df34 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:37:10.500659 20731 mvcc.cc:205] Tried to move back new op lower bound from 7332863305523458048 to 7332863305312817152. Current Snapshot: MvccSnapshot[applied={T|T < 7332863305523458048}]
I20260924 11:37:10.501646 20733 mvcc.cc:205] Tried to move back new op lower bound from 7332863305523458048 to 7332863305312817152. Current Snapshot: MvccSnapshot[applied={T|T < 7332863305523458048}]
I20260924 11:37:10.504858 20687 sys_catalog.cc:455] T 00000000000000000000000000000000 P f83a19c119bb4ae5a896674518cecf65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bcb864052b9743a582a95837c3f503db. Latest consensus state: current_term: 1 leader_uuid: "bcb864052b9743a582a95837c3f503db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcb864052b9743a582a95837c3f503db" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 34815 } } peers { permanent_uuid: "b89560fd87714b42876f8bbcd017df34" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 43399 } } peers { permanent_uuid: "f83a19c119bb4ae5a896674518cecf65" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 38045 } } }
I20260924 11:37:10.505218 20687 sys_catalog.cc:458] T 00000000000000000000000000000000 P f83a19c119bb4ae5a896674518cecf65 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:37:10.510218 20687 sys_catalog.cc:455] T 00000000000000000000000000000000 P f83a19c119bb4ae5a896674518cecf65 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bcb864052b9743a582a95837c3f503db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcb864052b9743a582a95837c3f503db" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 34815 } } peers { permanent_uuid: "b89560fd87714b42876f8bbcd017df34" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 43399 } } peers { permanent_uuid: "f83a19c119bb4ae5a896674518cecf65" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 38045 } } }
I20260924 11:37:10.510540 20687 sys_catalog.cc:458] T 00000000000000000000000000000000 P f83a19c119bb4ae5a896674518cecf65 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:37:10.511343 20728 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcb864052b9743a582a95837c3f503db [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bcb864052b9743a582a95837c3f503db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcb864052b9743a582a95837c3f503db" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 34815 } } peers { permanent_uuid: "b89560fd87714b42876f8bbcd017df34" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 43399 } } peers { permanent_uuid: "f83a19c119bb4ae5a896674518cecf65" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 38045 } } }
I20260924 11:37:10.511806 20728 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcb864052b9743a582a95837c3f503db [sys.catalog]: This master's current role is: LEADER
I20260924 11:37:10.513518 20730 catalog_manager.cc:1347] Generated new cluster ID: e102458466694be4815d64b89803c765
I20260924 11:37:10.513630 20730 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:37:10.527347 20739 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcb864052b9743a582a95837c3f503db [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bcb864052b9743a582a95837c3f503db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcb864052b9743a582a95837c3f503db" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 34815 } } peers { permanent_uuid: "b89560fd87714b42876f8bbcd017df34" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 43399 } } peers { permanent_uuid: "f83a19c119bb4ae5a896674518cecf65" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 38045 } } }
I20260924 11:37:10.527782 20739 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcb864052b9743a582a95837c3f503db [sys.catalog]: This master's current role is: LEADER
I20260924 11:37:10.528457 20715 sys_catalog.cc:455] T 00000000000000000000000000000000 P b89560fd87714b42876f8bbcd017df34 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bcb864052b9743a582a95837c3f503db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcb864052b9743a582a95837c3f503db" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 34815 } } peers { permanent_uuid: "b89560fd87714b42876f8bbcd017df34" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 43399 } } peers { permanent_uuid: "f83a19c119bb4ae5a896674518cecf65" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 38045 } } }
I20260924 11:37:10.528710 20715 sys_catalog.cc:458] T 00000000000000000000000000000000 P b89560fd87714b42876f8bbcd017df34 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:37:10.569509 20730 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:37:10.571437 20730 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:37:10.607826 20730 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bcb864052b9743a582a95837c3f503db: Generated new TSK 0
I20260924 11:37:10.609097 20730 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:37:11.147007 20698 catalog_manager.cc:1259] Loaded cluster ID: e102458466694be4815d64b89803c765
I20260924 11:37:11.147125 20698 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f83a19c119bb4ae5a896674518cecf65: loading cluster ID for follower catalog manager: success
I20260924 11:37:11.149824 20698 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f83a19c119bb4ae5a896674518cecf65: acquiring CA information for follower catalog manager: success
I20260924 11:37:11.151826 20698 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f83a19c119bb4ae5a896674518cecf65: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 11:37:11.314102 20726 catalog_manager.cc:1259] Loaded cluster ID: e102458466694be4815d64b89803c765
I20260924 11:37:11.314265 20726 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b89560fd87714b42876f8bbcd017df34: loading cluster ID for follower catalog manager: success
I20260924 11:37:11.316866 20726 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b89560fd87714b42876f8bbcd017df34: acquiring CA information for follower catalog manager: success
I20260924 11:37:11.318835 20726 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b89560fd87714b42876f8bbcd017df34: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 11:37:11.765182 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 20459
I20260924 11:37:11.766348 20521 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:37:11.883333 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 20459
I20260924 11:37:11.906569 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 20533
I20260924 11:37:11.907933 20595 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:37:12.018213 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 20533
I20260924 11:37:12.037782 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 20607
I20260924 11:37:12.039225 20669 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:37:12.150669 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 20607
2026-09-24T11:37:12Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5011 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790249617057452-6992-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:37:12 dist-test-slave-cq7c krb5kdc[20751](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:37:12 dist-test-slave-cq7c krb5kdc[20751](info): set up 2 sockets
Sep 24 11:37:12 dist-test-slave-cq7c krb5kdc[20751](info): commencing operation
krb5kdc: starting...
W20260924 11:37:13.292075 6992 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Sep 24 11:37:13 dist-test-slave-cq7c krb5kdc[20751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:37:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:37:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.212.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
I20260924 11:37:13.558913 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:42685
--webserver_interface=127.6.212.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:40865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:42685
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:37:13.908953 20764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:37:13.909358 20764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:37:13.909531 20764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:37:13.917303 20764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:37:13.917488 20764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:37:13.917603 20764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:37:13.917708 20764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:37:13.917783 20764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:37:13.928282 20764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:40865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:42685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:42685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.212.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:37:13.929805 20764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:37:13.931209 20764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:37:13.944393 20769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:37:13.944694 20770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:37:13.945489 20772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:37:13.946650 20764 server_base.cc:1034] running on GCE node
I20260924 11:37:13.947573 20764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:37:13.949179 20764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:37:13.950415 20764 hybrid_clock.cc:648] HybridClock initialized: now 1790249833950362 us; error 49 us; skew 500 ppm
I20260924 11:37:13.950820 20764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:37:13 dist-test-slave-cq7c krb5kdc[20751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249833, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:37:13.958067 20764 init.cc:377] Logged in from keytab as kudu/127.6.212.62@KRBTEST.COM (short username kudu)
I20260924 11:37:13.959982 20764 webserver.cc:460] Webserver started at http://127.6.212.62:41363/ using document root <none> and password file <none>
I20260924 11:37:13.960747 20764 fs_manager.cc:359] Metadata directory not provided
I20260924 11:37:13.960853 20764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:37:13.961127 20764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:37:13.964349 20764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "8ef6cfe1793f408e88a84f1481fb4561"
format_stamp: "Formatted at 2026-09-24 11:37:13 on dist-test-slave-cq7c"
server_key: "1c1f9b98ecdda3221e1fe5b0e21fd44d"
server_key_iv: "6bb23d4fc1f2ed9bf299456601b12b04"
server_key_version: "encryptionkey@0"
I20260924 11:37:13.965119 20764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "8ef6cfe1793f408e88a84f1481fb4561"
format_stamp: "Formatted at 2026-09-24 11:37:13 on dist-test-slave-cq7c"
server_key: "1c1f9b98ecdda3221e1fe5b0e21fd44d"
server_key_iv: "6bb23d4fc1f2ed9bf299456601b12b04"
server_key_version: "encryptionkey@0"
I20260924 11:37:13.970711 20764 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 11:37:13.974781 20779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:13.976462 20764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:37:13.976713 20764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "8ef6cfe1793f408e88a84f1481fb4561"
format_stamp: "Formatted at 2026-09-24 11:37:13 on dist-test-slave-cq7c"
server_key: "1c1f9b98ecdda3221e1fe5b0e21fd44d"
server_key_iv: "6bb23d4fc1f2ed9bf299456601b12b04"
server_key_version: "encryptionkey@0"
I20260924 11:37:13.976964 20764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:37:13.998469 20764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:37:14.003034 20764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:37:14.003381 20764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:37:14.023531 20764 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.62:42685
I20260924 11:37:14.023607 20830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.62:42685 every 8 connection(s)
I20260924 11:37:14.025182 20764 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:37:14.029649 20831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:37:14.034191 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 20764
I20260924 11:37:14.034433 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:37:14.035120 6992 external_mini_cluster.cc:1428] Setting key 3635b1b2c6f789083435cf9ac835fe67
I20260924 11:37:14.043025 20831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ef6cfe1793f408e88a84f1481fb4561: Bootstrap starting.
I20260924 11:37:14.047322 20831 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ef6cfe1793f408e88a84f1481fb4561: Neither blocks nor log segments found. Creating new log.
I20260924 11:37:14.049186 20831 log.cc:824] T 00000000000000000000000000000000 P 8ef6cfe1793f408e88a84f1481fb4561: Log is configured to *not* fsync() on all Append() calls
Sep 24 11:37:14 dist-test-slave-cq7c krb5kdc[20751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:37:14.053007 20831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ef6cfe1793f408e88a84f1481fb4561: No bootstrap required, opened a new log
I20260924 11:37:14.062222 20834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:14.037415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40138 (local address 127.6.212.62:42685)
0924 11:37:14.038263 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:14.038275 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:14.038319 (+ 44us) server_negotiation.cc:407] Connection header received
0924 11:37:14.040973 (+ 2654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:14.041004 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:14.041476 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:14.041879 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:14.044102 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:14.045527 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:14.047806 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:14.048589 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:14.049006 (+ 417us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:14.054042 (+ 5036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:14.054051 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:14.054068 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:14.054115 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:14.057554 (+ 3439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:14.058429 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:14.058436 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:14.058443 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:14.058630 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:14.059058 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:14.059065 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:14.059067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:14.059778 (+ 711us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:14.060076 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:14.060614 (+ 538us) server_negotiation.cc:299] Negotiation successful
0924 11:37:14.061109 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":387,"threads_started":1}
I20260924 11:37:14.062032 20831 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ef6cfe1793f408e88a84f1481fb4561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ef6cfe1793f408e88a84f1481fb4561" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42685 } }
I20260924 11:37:14.062491 20831 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ef6cfe1793f408e88a84f1481fb4561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:37:14.062604 20831 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ef6cfe1793f408e88a84f1481fb4561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ef6cfe1793f408e88a84f1481fb4561, State: Initialized, Role: FOLLOWER
I20260924 11:37:14.063266 20831 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ef6cfe1793f408e88a84f1481fb4561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ef6cfe1793f408e88a84f1481fb4561" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42685 } }
I20260924 11:37:14.063678 20831 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ef6cfe1793f408e88a84f1481fb4561 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:37:14.063849 20831 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ef6cfe1793f408e88a84f1481fb4561 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:37:14.064039 20831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ef6cfe1793f408e88a84f1481fb4561 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:37:14.066572 6992 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:37:14.067164 20831 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ef6cfe1793f408e88a84f1481fb4561 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ef6cfe1793f408e88a84f1481fb4561" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42685 } }
I20260924 11:37:14.067811 20831 leader_election.cc:302] T 00000000000000000000000000000000 P 8ef6cfe1793f408e88a84f1481fb4561 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ef6cfe1793f408e88a84f1481fb4561; no voters:
I20260924 11:37:14.068454 20831 leader_election.cc:288] T 00000000000000000000000000000000 P 8ef6cfe1793f408e88a84f1481fb4561 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:37:14.068572 20836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ef6cfe1793f408e88a84f1481fb4561 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:37:14.070859 20831 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ef6cfe1793f408e88a84f1481fb4561 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:37:14.070915 20836 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ef6cfe1793f408e88a84f1481fb4561 [term 1 LEADER]: Becoming Leader. State: Replica: 8ef6cfe1793f408e88a84f1481fb4561, State: Running, Role: LEADER
I20260924 11:37:14.071841 20836 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ef6cfe1793f408e88a84f1481fb4561 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ef6cfe1793f408e88a84f1481fb4561" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42685 } }
I20260924 11:37:14.075626 20838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ef6cfe1793f408e88a84f1481fb4561 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ef6cfe1793f408e88a84f1481fb4561. Latest consensus state: current_term: 1 leader_uuid: "8ef6cfe1793f408e88a84f1481fb4561" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ef6cfe1793f408e88a84f1481fb4561" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42685 } } }
I20260924 11:37:14.076057 20838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ef6cfe1793f408e88a84f1481fb4561 [sys.catalog]: This master's current role is: LEADER
I20260924 11:37:14.075168 20837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ef6cfe1793f408e88a84f1481fb4561 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ef6cfe1793f408e88a84f1481fb4561" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ef6cfe1793f408e88a84f1481fb4561" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 42685 } } }
I20260924 11:37:14.079742 20837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ef6cfe1793f408e88a84f1481fb4561 [sys.catalog]: This master's current role is: LEADER
I20260924 11:37:14.083005 20849 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:37:14.088053 20849 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:37:14.097193 20849 catalog_manager.cc:1347] Generated new cluster ID: 3d3e69cee5844593a264a4b361f87e25
I20260924 11:37:14.097338 20849 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:37:14.107175 20849 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:37:14.109283 20849 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:37:14.119946 20849 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ef6cfe1793f408e88a84f1481fb4561: Generated new TSK 0
I20260924 11:37:14.121132 20849 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:37:14.180413 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 20764
I20260924 11:37:14.181748 20826 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:37:14.280808 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 20764
2026-09-24T11:37:14Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2123 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:37:14 dist-test-slave-cq7c krb5kdc[20860](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:37:14 dist-test-slave-cq7c krb5kdc[20860](info): set up 2 sockets
Sep 24 11:37:14 dist-test-slave-cq7c krb5kdc[20860](info): commencing operation
krb5kdc: starting...
W20260924 11:37:15.399894 6992 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Sep 24 11:37:15 dist-test-slave-cq7c krb5kdc[20860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:37:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:37:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.212.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
I20260924 11:37:15.668166 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:37655
--webserver_interface=127.6.212.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:36593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:37655
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:37:16.041275 20873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:37:16.041687 20873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:37:16.041862 20873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:37:16.049633 20873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:37:16.049832 20873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:37:16.049934 20873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:37:16.050027 20873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:37:16.050100 20873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:37:16.060557 20873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:36593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:37655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:37655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.212.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:37:16.062146 20873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:37:16.063747 20873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:37:16.077971 20878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:37:16.077940 20881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:37:16.077920 20879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:37:16.079051 20873 server_base.cc:1034] running on GCE node
I20260924 11:37:16.080034 20873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:37:16.081676 20873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:37:16.083063 20873 hybrid_clock.cc:648] HybridClock initialized: now 1790249836082964 us; error 63 us; skew 500 ppm
I20260924 11:37:16.083592 20873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:37:16 dist-test-slave-cq7c krb5kdc[20860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249836, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:37:16.087613 20873 init.cc:377] Logged in from keytab as kudu/127.6.212.62@KRBTEST.COM (short username kudu)
I20260924 11:37:16.089193 20873 webserver.cc:460] Webserver started at http://127.6.212.62:37835/ using document root <none> and password file <none>
I20260924 11:37:16.089955 20873 fs_manager.cc:359] Metadata directory not provided
I20260924 11:37:16.090087 20873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:37:16.090412 20873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:37:16.093619 20873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "c9d70886d7b64358958769d14824cdbd"
format_stamp: "Formatted at 2026-09-24 11:37:16 on dist-test-slave-cq7c"
server_key: "c8bfd4e83afd1a56d29bbe80e0d24a98"
server_key_iv: "60f6236b101981a12fbd31c87a8258e3"
server_key_version: "encryptionkey@0"
I20260924 11:37:16.094425 20873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "c9d70886d7b64358958769d14824cdbd"
format_stamp: "Formatted at 2026-09-24 11:37:16 on dist-test-slave-cq7c"
server_key: "c8bfd4e83afd1a56d29bbe80e0d24a98"
server_key_iv: "60f6236b101981a12fbd31c87a8258e3"
server_key_version: "encryptionkey@0"
I20260924 11:37:16.100114 20873 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 11:37:16.104043 20888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:16.105612 20873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:37:16.105839 20873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "c9d70886d7b64358958769d14824cdbd"
format_stamp: "Formatted at 2026-09-24 11:37:16 on dist-test-slave-cq7c"
server_key: "c8bfd4e83afd1a56d29bbe80e0d24a98"
server_key_iv: "60f6236b101981a12fbd31c87a8258e3"
server_key_version: "encryptionkey@0"
I20260924 11:37:16.106072 20873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:37:16.126132 20873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:37:16.131170 20873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:37:16.131528 20873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:37:16.152403 20873 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.62:37655
I20260924 11:37:16.152422 20939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.62:37655 every 8 connection(s)
I20260924 11:37:16.153945 20873 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:37:16.154417 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 20873
I20260924 11:37:16.154621 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:37:16.155588 6992 external_mini_cluster.cc:1428] Setting key e295fec210d7307cf8b194aacaf860b2
I20260924 11:37:16.158721 20940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 11:37:16 dist-test-slave-cq7c krb5kdc[20860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:37:16.171046 20940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd: Bootstrap starting.
I20260924 11:37:16.174868 20940 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd: Neither blocks nor log segments found. Creating new log.
I20260924 11:37:16.176095 20940 log.cc:824] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd: Log is configured to *not* fsync() on all Append() calls
I20260924 11:37:16.178619 20943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:16.157698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35994 (local address 127.6.212.62:37655)
0924 11:37:16.158528 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:16.158541 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:16.158581 (+ 40us) server_negotiation.cc:407] Connection header received
0924 11:37:16.160329 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:16.160352 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:16.160761 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:16.161159 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:16.162642 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:16.164145 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:16.165643 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:16.166338 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:16.166549 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:16.171006 (+ 4457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:16.171014 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:16.171034 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:16.171078 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:16.174363 (+ 3285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:16.175131 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:16.175138 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:16.175144 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:16.175321 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:16.175757 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:16.175762 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:16.175765 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:16.176361 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:16.176665 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:16.177173 (+ 508us) server_negotiation.cc:299] Negotiation successful
0924 11:37:16.177614 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":375,"threads_started":1}
I20260924 11:37:16.179352 20940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd: No bootstrap required, opened a new log
I20260924 11:37:16.188576 20940 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d70886d7b64358958769d14824cdbd" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 37655 } }
I20260924 11:37:16.189069 20940 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:37:16.189210 20940 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9d70886d7b64358958769d14824cdbd, State: Initialized, Role: FOLLOWER
I20260924 11:37:16.189842 20940 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d70886d7b64358958769d14824cdbd" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 37655 } }
I20260924 11:37:16.190133 20940 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:37:16.190289 20940 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:37:16.190481 20940 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:37:16.193476 20940 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d70886d7b64358958769d14824cdbd" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 37655 } }
I20260924 11:37:16.194054 20940 leader_election.cc:302] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9d70886d7b64358958769d14824cdbd; no voters:
I20260924 11:37:16.194703 20940 leader_election.cc:288] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:37:16.194828 20945 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:37:16.195628 20945 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [term 1 LEADER]: Becoming Leader. State: Replica: c9d70886d7b64358958769d14824cdbd, State: Running, Role: LEADER
I20260924 11:37:16.196251 20945 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d70886d7b64358958769d14824cdbd" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 37655 } }
I20260924 11:37:16.197135 20940 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:37:16.199534 20947 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9d70886d7b64358958769d14824cdbd. Latest consensus state: current_term: 1 leader_uuid: "c9d70886d7b64358958769d14824cdbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d70886d7b64358958769d14824cdbd" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 37655 } } }
I20260924 11:37:16.199617 20946 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9d70886d7b64358958769d14824cdbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d70886d7b64358958769d14824cdbd" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 37655 } } }
I20260924 11:37:16.200258 20947 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [sys.catalog]: This master's current role is: LEADER
I20260924 11:37:16.200277 20946 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [sys.catalog]: This master's current role is: LEADER
I20260924 11:37:16.202958 20953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:37:16.208091 20953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:37:16.217893 20953 catalog_manager.cc:1347] Generated new cluster ID: dd49dfdc53b54a94bb35fde2787bbb5c
I20260924 11:37:16.218039 20953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:37:16.234272 20953 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:37:16.236372 20953 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:37:16.251726 20953 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd: Generated new TSK 0
I20260924 11:37:16.252723 20953 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.212.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab.
Entry for principal kudu/127.6.212.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab.
I20260924 11:37:16.328521 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.1:0
--local_ip_for_outbound_sockets=127.6.212.1
--webserver_interface=127.6.212.1
--webserver_port=0
--tserver_master_addrs=127.6.212.62:37655
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab
--principal=kudu/127.6.212.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:36593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:37:16.683691 20966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:37:16.684191 20966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:37:16.684340 20966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:37:16.691984 20966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:37:16.692128 20966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:37:16.692303 20966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.1
I20260924 11:37:16.703271 20966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:36593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab
--principal=kudu/127.6.212.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.212.1
--webserver_port=0
--tserver_master_addrs=127.6.212.62:37655
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.1
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:37:16.704804 20966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:37:16.706284 20966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:37:16.720814 20971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:37:16.720986 20974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:37:16.721833 20972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:37:16.721823 20966 server_base.cc:1034] running on GCE node
I20260924 11:37:16.722625 20966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:37:16.723769 20966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:37:16.725021 20966 hybrid_clock.cc:648] HybridClock initialized: now 1790249836724970 us; error 51 us; skew 500 ppm
I20260924 11:37:16.725363 20966 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:37:16 dist-test-slave-cq7c krb5kdc[20860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249836, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:37:16.728750 20966 init.cc:377] Logged in from keytab as kudu/127.6.212.1@KRBTEST.COM (short username kudu)
I20260924 11:37:16.730432 20966 webserver.cc:460] Webserver started at http://127.6.212.1:38555/ using document root <none> and password file <none>
I20260924 11:37:16.731208 20966 fs_manager.cc:359] Metadata directory not provided
I20260924 11:37:16.731385 20966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:37:16.731702 20966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:37:16.734866 20966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-0/data/instance:
uuid: "bbffa38f702a4e08b829ed069a47c103"
format_stamp: "Formatted at 2026-09-24 11:37:16 on dist-test-slave-cq7c"
server_key: "2a7fcf6e99679b71e950e59b74a52b03"
server_key_iv: "d6364fb283b253faa7a850615bff5b73"
server_key_version: "encryptionkey@0"
I20260924 11:37:16.735716 20966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance:
uuid: "bbffa38f702a4e08b829ed069a47c103"
format_stamp: "Formatted at 2026-09-24 11:37:16 on dist-test-slave-cq7c"
server_key: "2a7fcf6e99679b71e950e59b74a52b03"
server_key_iv: "d6364fb283b253faa7a850615bff5b73"
server_key_version: "encryptionkey@0"
I20260924 11:37:16.741307 20966 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 11:37:16.745575 20981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:16.747007 20966 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 11:37:16.747187 20966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-0/wal
uuid: "bbffa38f702a4e08b829ed069a47c103"
format_stamp: "Formatted at 2026-09-24 11:37:16 on dist-test-slave-cq7c"
server_key: "2a7fcf6e99679b71e950e59b74a52b03"
server_key_iv: "d6364fb283b253faa7a850615bff5b73"
server_key_version: "encryptionkey@0"
I20260924 11:37:16.747429 20966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:37:16.764794 20966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:37:16.769471 20966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:37:16.769776 20966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:37:16.770844 20966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:37:16.772430 20966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:37:16.772535 20966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:16.772679 20966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:37:16.772750 20966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:16.808084 20966 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.1:40045
I20260924 11:37:16.808182 21093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.1:40045 every 8 connection(s)
I20260924 11:37:16.809603 20966 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
I20260924 11:37:16.816365 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 20966
I20260924 11:37:16.816622 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance
I20260924 11:37:16.817286 6992 external_mini_cluster.cc:1428] Setting key 0055e544b34db15bc37acfb15e8f0129
Sep 24 11:37:16 dist-test-slave-cq7c krb5kdc[20860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249836, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:37:16.831094 21096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:16.812509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:43707 (local address 127.6.212.62:37655)
0924 11:37:16.812993 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:16.812997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:16.813920 (+ 923us) server_negotiation.cc:407] Connection header received
0924 11:37:16.815943 (+ 2023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:16.815949 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:16.816058 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:16.816179 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:16.817983 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:16.818736 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:16.820474 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:16.820894 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:16.821052 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:16.824854 (+ 3802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:16.824860 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:16.824863 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:16.824900 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:16.827635 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:16.828504 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:16.828509 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:16.828512 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:16.828642 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:16.829117 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:16.829121 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:16.829124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:16.829408 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:16.829573 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:16.830427 (+ 854us) server_negotiation.cc:299] Negotiation successful
0924 11:37:16.830684 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":259,"threads_started":1}
I20260924 11:37:16.832221 21097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:16.812991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.1:43707)
0924 11:37:16.813750 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:16.813797 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 11:37:16.815633 (+ 1836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:37:16.816488 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:37:16.816502 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:37:16.817124 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:37:16.817782 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:16.817808 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:16.818932 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:16.818936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:16.820307 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:16.820321 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:16.821155 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:16.821163 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:16.821446 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:16.822226 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:37:16.822250 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:37:16.824574 (+ 2324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:37:16.827824 (+ 3250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:16.827835 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:16.827858 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:16.828347 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:16.828761 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:16.828766 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:16.828770 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:16.828987 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:16.829546 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:37:16.829556 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:37:16.830018 (+ 462us) client_negotiation.cc:769] Sending connection context
0924 11:37:16.830315 (+ 297us) client_negotiation.cc:240] Negotiation successful
0924 11:37:16.830944 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":285,"threads_started":1}
I20260924 11:37:16.834105 21094 heartbeater.cc:347] Connected to a master server at 127.6.212.62:37655
I20260924 11:37:16.834498 21094 heartbeater.cc:464] Registering TS with master...
I20260924 11:37:16.835563 21094 heartbeater.cc:511] Master 127.6.212.62:37655 requested a full tablet report, sending...
I20260924 11:37:16.838469 20905 ts_manager.cc:194] Registered new tserver with Master: bbffa38f702a4e08b829ed069a47c103 (127.6.212.1:40045)
I20260924 11:37:16.840878 20905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.1@KRBTEST.COM'} at 127.6.212.1:43707
WARNING: no policy specified for kudu/127.6.212.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab.
Entry for principal kudu/127.6.212.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab.
I20260924 11:37:16.878690 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.2:0
--local_ip_for_outbound_sockets=127.6.212.2
--webserver_interface=127.6.212.2
--webserver_port=0
--tserver_master_addrs=127.6.212.62:37655
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab
--principal=kudu/127.6.212.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:36593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:37:17.226995 21100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:37:17.227306 21100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:37:17.227476 21100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:37:17.235111 21100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:37:17.235239 21100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:37:17.235402 21100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.2
I20260924 11:37:17.245570 21100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:36593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab
--principal=kudu/127.6.212.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.212.2
--webserver_port=0
--tserver_master_addrs=127.6.212.62:37655
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.2
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:37:17.247014 21100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:37:17.248356 21100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:37:17.262079 21105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:37:17.262005 21108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:37:17.262071 21106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:37:17.263028 21100 server_base.cc:1034] running on GCE node
I20260924 11:37:17.263733 21100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:37:17.264817 21100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:37:17.266112 21100 hybrid_clock.cc:648] HybridClock initialized: now 1790249837266037 us; error 65 us; skew 500 ppm
I20260924 11:37:17.266523 21100 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:37:17 dist-test-slave-cq7c krb5kdc[20860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249837, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:37:17.269928 21100 init.cc:377] Logged in from keytab as kudu/127.6.212.2@KRBTEST.COM (short username kudu)
I20260924 11:37:17.271529 21100 webserver.cc:460] Webserver started at http://127.6.212.2:35125/ using document root <none> and password file <none>
I20260924 11:37:17.272249 21100 fs_manager.cc:359] Metadata directory not provided
I20260924 11:37:17.272349 21100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:37:17.272616 21100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:37:17.275722 21100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-1/data/instance:
uuid: "6cb7f6130bda4786bc162352b37ee6ca"
format_stamp: "Formatted at 2026-09-24 11:37:17 on dist-test-slave-cq7c"
server_key: "e1642fcd032ee8456f94393532c660ee"
server_key_iv: "6bc1321ed194db0a5bc1e8395b4f0829"
server_key_version: "encryptionkey@0"
I20260924 11:37:17.276460 21100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance:
uuid: "6cb7f6130bda4786bc162352b37ee6ca"
format_stamp: "Formatted at 2026-09-24 11:37:17 on dist-test-slave-cq7c"
server_key: "e1642fcd032ee8456f94393532c660ee"
server_key_iv: "6bc1321ed194db0a5bc1e8395b4f0829"
server_key_version: "encryptionkey@0"
I20260924 11:37:17.281687 21100 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 11:37:17.285181 21115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:17.286669 21100 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 11:37:17.286940 21100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-1/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-1/wal
uuid: "6cb7f6130bda4786bc162352b37ee6ca"
format_stamp: "Formatted at 2026-09-24 11:37:17 on dist-test-slave-cq7c"
server_key: "e1642fcd032ee8456f94393532c660ee"
server_key_iv: "6bc1321ed194db0a5bc1e8395b4f0829"
server_key_version: "encryptionkey@0"
I20260924 11:37:17.287158 21100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:37:17.314401 21100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:37:17.319231 21100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:37:17.319619 21100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:37:17.320864 21100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:37:17.322178 21100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:37:17.322256 21100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:17.322348 21100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:37:17.322395 21100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:17.358024 21100 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.2:43247
I20260924 11:37:17.358142 21227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.2:43247 every 8 connection(s)
I20260924 11:37:17.359431 21100 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
I20260924 11:37:17.364934 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 21100
I20260924 11:37:17.365134 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance
I20260924 11:37:17.365684 6992 external_mini_cluster.cc:1428] Setting key cb4e05e72904c26f45be131f18ec4ac4
Sep 24 11:37:17 dist-test-slave-cq7c krb5kdc[20860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249837, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.2@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:37:17.384488 21230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:17.362157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:44707 (local address 127.6.212.62:37655)
0924 11:37:17.362639 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:17.362643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:17.363586 (+ 943us) server_negotiation.cc:407] Connection header received
0924 11:37:17.365949 (+ 2363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:17.365953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:17.366057 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:17.366156 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:17.367903 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:17.368626 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:17.371345 (+ 2719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:17.371958 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:17.372086 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:17.377072 (+ 4986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:17.377081 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:17.377089 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:17.377139 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:17.380931 (+ 3792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:17.381720 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:17.381728 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:17.381735 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:17.381915 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:17.382327 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:17.382334 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:17.382340 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:17.382726 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:17.382890 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:17.383673 (+ 783us) server_negotiation.cc:299] Negotiation successful
0924 11:37:17.383982 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":253,"threads_started":1}
I20260924 11:37:17.385093 21231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:17.362604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.2:44707)
0924 11:37:17.363423 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:17.363468 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 11:37:17.365623 (+ 2155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:37:17.366434 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:37:17.366443 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:37:17.367017 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:37:17.367694 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:17.367712 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:17.368794 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:17.368802 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:17.371149 (+ 2347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:17.371169 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:17.372236 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:17.372246 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:17.372606 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:17.373447 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:37:17.373486 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:37:17.376705 (+ 3219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:37:17.381089 (+ 4384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:17.381096 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:17.381112 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:17.381570 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:17.382022 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:17.382025 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:17.382028 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:17.382218 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:17.382843 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:37:17.382849 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:37:17.383230 (+ 381us) client_negotiation.cc:769] Sending connection context
0924 11:37:17.383520 (+ 290us) client_negotiation.cc:240] Negotiation successful
0924 11:37:17.383936 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":344,"threads_started":1}
I20260924 11:37:17.386739 21228 heartbeater.cc:347] Connected to a master server at 127.6.212.62:37655
I20260924 11:37:17.387091 21228 heartbeater.cc:464] Registering TS with master...
I20260924 11:37:17.387871 21228 heartbeater.cc:511] Master 127.6.212.62:37655 requested a full tablet report, sending...
I20260924 11:37:17.389485 20905 ts_manager.cc:194] Registered new tserver with Master: 6cb7f6130bda4786bc162352b37ee6ca (127.6.212.2:43247)
I20260924 11:37:17.390326 20905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.2@KRBTEST.COM'} at 127.6.212.2:44707
WARNING: no policy specified for kudu/127.6.212.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab.
Entry for principal kudu/127.6.212.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab.
I20260924 11:37:17.426692 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.3:0
--local_ip_for_outbound_sockets=127.6.212.3
--webserver_interface=127.6.212.3
--webserver_port=0
--tserver_master_addrs=127.6.212.62:37655
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab
--principal=kudu/127.6.212.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:36593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:37:17.792212 21234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:37:17.792544 21234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:37:17.792769 21234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:37:17.801512 21234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:37:17.801657 21234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:37:17.801926 21234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.3
I20260924 11:37:17.813922 21234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:36593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab
--principal=kudu/127.6.212.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.212.3
--webserver_port=0
--tserver_master_addrs=127.6.212.62:37655
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.3
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:37:17.815423 21234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:37:17.816920 21234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:37:17.831701 21240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:37:17.831640 21242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:37:17.831845 21239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:37:17.832715 21234 server_base.cc:1034] running on GCE node
I20260924 11:37:17.833362 21234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:37:17.834331 21234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:37:17.835566 21234 hybrid_clock.cc:648] HybridClock initialized: now 1790249837835511 us; error 43 us; skew 500 ppm
I20260924 11:37:17.835896 21234 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:37:17 dist-test-slave-cq7c krb5kdc[20860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249837, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:37:17.839084 21234 init.cc:377] Logged in from keytab as kudu/127.6.212.3@KRBTEST.COM (short username kudu)
I20260924 11:37:17.840703 21234 webserver.cc:460] Webserver started at http://127.6.212.3:43683/ using document root <none> and password file <none>
I20260924 11:37:17.841419 21234 fs_manager.cc:359] Metadata directory not provided
I20260924 11:37:17.841566 21234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:37:17.841877 21234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:37:17.844904 21094 heartbeater.cc:503] Master 127.6.212.62:37655 was elected leader, sending a full tablet report...
I20260924 11:37:17.844906 21234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-2/data/instance:
uuid: "1d695defd9734c80a12887ed84ede67c"
format_stamp: "Formatted at 2026-09-24 11:37:17 on dist-test-slave-cq7c"
server_key: "70770c8ecb3d8f8f6046c7648a791921"
server_key_iv: "dac51743039c1ff5ea727efd5dcac7ee"
server_key_version: "encryptionkey@0"
I20260924 11:37:17.845827 21234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance:
uuid: "1d695defd9734c80a12887ed84ede67c"
format_stamp: "Formatted at 2026-09-24 11:37:17 on dist-test-slave-cq7c"
server_key: "70770c8ecb3d8f8f6046c7648a791921"
server_key_iv: "dac51743039c1ff5ea727efd5dcac7ee"
server_key_version: "encryptionkey@0"
I20260924 11:37:17.851652 21234 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 11:37:17.855288 21249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:17.856946 21234 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:37:17.857125 21234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-2/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-2/wal
uuid: "1d695defd9734c80a12887ed84ede67c"
format_stamp: "Formatted at 2026-09-24 11:37:17 on dist-test-slave-cq7c"
server_key: "70770c8ecb3d8f8f6046c7648a791921"
server_key_iv: "dac51743039c1ff5ea727efd5dcac7ee"
server_key_version: "encryptionkey@0"
I20260924 11:37:17.857338 21234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:37:17.880280 21234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:37:17.884915 21234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:37:17.885232 21234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:37:17.886268 21234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:37:17.887844 21234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:37:17.887955 21234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:17.888403 21234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:37:17.888500 21234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:17.923062 21234 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.3:44015
I20260924 11:37:17.923134 21361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.3:44015 every 8 connection(s)
I20260924 11:37:17.924491 21234 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
I20260924 11:37:17.931663 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 21234
I20260924 11:37:17.931849 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance
I20260924 11:37:17.932300 6992 external_mini_cluster.cc:1428] Setting key 5a5d26a4e117a5a54a6ced4ea053330b
Sep 24 11:37:17 dist-test-slave-cq7c krb5kdc[20860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249837, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.3@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:37:17.945576 21364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:17.927131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:50091 (local address 127.6.212.62:37655)
0924 11:37:17.927644 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:17.927649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:17.928556 (+ 907us) server_negotiation.cc:407] Connection header received
0924 11:37:17.930436 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:17.930454 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:17.930561 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:17.930683 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:17.932544 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:17.933487 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:17.935413 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:17.935916 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:17.936017 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:17.939567 (+ 3550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:17.939574 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:17.939577 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:17.939616 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:17.942308 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:17.943071 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:17.943076 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:17.943078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:17.943232 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:17.943706 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:17.943710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:17.943712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:17.944079 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:17.944231 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:17.945068 (+ 837us) server_negotiation.cc:299] Negotiation successful
0924 11:37:17.945251 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":279,"threads_started":1}
I20260924 11:37:17.946712 21365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:17.927594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.3:50091)
0924 11:37:17.928367 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:17.928422 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 11:37:17.930147 (+ 1725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:37:17.930931 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:37:17.930947 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:37:17.931586 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:37:17.932291 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:17.932314 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:17.933643 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:17.933647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:17.935199 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:17.935211 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:17.936112 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:17.936117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:17.936315 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:17.937005 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:37:17.937029 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:37:17.939265 (+ 2236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:37:17.942440 (+ 3175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:17.942457 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:17.942472 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:17.942902 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:17.943343 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:17.943346 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:17.943348 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:17.943587 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:17.944211 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:37:17.944217 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:37:17.944550 (+ 333us) client_negotiation.cc:769] Sending connection context
0924 11:37:17.944903 (+ 353us) client_negotiation.cc:240] Negotiation successful
0924 11:37:17.945357 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":354,"threads_started":1}
I20260924 11:37:17.948598 21362 heartbeater.cc:347] Connected to a master server at 127.6.212.62:37655
I20260924 11:37:17.948933 21362 heartbeater.cc:464] Registering TS with master...
I20260924 11:37:17.949674 21362 heartbeater.cc:511] Master 127.6.212.62:37655 requested a full tablet report, sending...
I20260924 11:37:17.951232 20905 ts_manager.cc:194] Registered new tserver with Master: 1d695defd9734c80a12887ed84ede67c (127.6.212.3:44015)
I20260924 11:37:17.952427 20905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.3@KRBTEST.COM'} at 127.6.212.3:50091
I20260924 11:37:17.960888 6992 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:37:17.978506 21364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:17.965820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33558 (local address 127.6.212.62:37655)
0924 11:37:17.966160 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:17.966167 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:17.966570 (+ 403us) server_negotiation.cc:407] Connection header received
0924 11:37:17.966786 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:17.966792 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:17.966893 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:17.967000 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:17.968523 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:17.969237 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:17.970684 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:17.971100 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:17.971195 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:17.973531 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:17.973535 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:17.973539 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:17.973571 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:17.976192 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:17.976869 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:17.976873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:17.976875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:17.976984 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:17.977299 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:17.977302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:17.977304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:17.977618 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:17.977796 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:17.978092 (+ 296us) server_negotiation.cc:299] Negotiation successful
0924 11:37:17.978253 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 11:37:17.982582 20905 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33558:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:37:17.989712 20905 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:37:18.049281 21375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:18.002546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:40045 (local address 127.0.0.1:46636)
0924 11:37:18.040974 (+ 38428us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:18.041003 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 11:37:18.041256 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:37:18.042172 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:37:18.042177 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:37:18.042219 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:37:18.042961 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:18.042969 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:18.044819 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:18.044825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:18.046769 (+ 1944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:18.046782 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:18.048465 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:18.048469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:18.048602 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:18.048637 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 11:37:18.048748 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 11:37:18.048885 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38267,"mutex_wait_us":110,"spinlock_wait_cycles":4736,"thread_start_us":35907,"threads_started":1}
I20260924 11:37:18.049322 21380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:18.003296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46636 (local address 127.6.212.1:40045)
0924 11:37:18.036448 (+ 33152us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:18.036463 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:18.041135 (+ 4672us) server_negotiation.cc:407] Connection header received
0924 11:37:18.041491 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:18.041498 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:18.041683 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:18.041885 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:37:18.043137 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:18.044694 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:18.046926 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:18.048283 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:18.048415 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:18.048679 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:18.048789 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 11:37:18.048957 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32862,"thread_start_us":33147,"threads_started":1}
I20260924 11:37:18.050209 21377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:18.002484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.2:43247 (local address 127.0.0.1:36896)
0924 11:37:18.040895 (+ 38411us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:18.040926 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 11:37:18.041171 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:37:18.042172 (+ 1001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:37:18.042177 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:37:18.042212 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:37:18.042942 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:18.042956 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:18.044946 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:18.044950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:18.047504 (+ 2554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:18.047519 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:18.049294 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:18.049304 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:18.049495 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:18.049526 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 11:37:18.049615 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 11:37:18.049736 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38233,"mutex_wait_us":121,"spinlock_wait_cycles":896,"thread_start_us":37634,"threads_started":1}
I20260924 11:37:18.050915 21379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:18.002908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36896 (local address 127.6.212.2:43247)
0924 11:37:18.039873 (+ 36965us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:18.039887 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:18.041042 (+ 1155us) server_negotiation.cc:407] Connection header received
0924 11:37:18.041368 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:18.041375 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:18.041679 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:18.041885 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:37:18.043166 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:18.044759 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:18.047660 (+ 2901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:18.049069 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:18.049212 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:18.049400 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:18.050432 (+ 1032us) server_negotiation.cc:299] Negotiation successful
0924 11:37:18.050592 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36706,"thread_start_us":36540,"threads_started":1}
I20260924 11:37:18.051951 21378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:18.001923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.3:44015 (local address 127.0.0.1:37384)
0924 11:37:18.037799 (+ 35876us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:18.037886 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0924 11:37:18.038161 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:37:18.041942 (+ 3781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:37:18.041950 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:37:18.041980 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:37:18.042530 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:18.042547 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:18.044582 (+ 2035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:18.044587 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:18.047504 (+ 2917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:18.047519 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:18.051041 (+ 3522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:18.051049 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:18.051228 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:18.051264 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 11:37:18.051431 (+ 167us) client_negotiation.cc:240] Negotiation successful
0924 11:37:18.051570 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35490,"mutex_wait_us":83,"thread_start_us":38609,"threads_started":1}
I20260924 11:37:18.052748 21376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:18.002130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37384 (local address 127.6.212.3:44015)
0924 11:37:18.041278 (+ 39148us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:18.041288 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:18.041320 (+ 32us) server_negotiation.cc:407] Connection header received
0924 11:37:18.041421 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:18.041431 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:18.041656 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:18.041933 (+ 277us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:37:18.042745 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:18.044463 (+ 1718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:18.049491 (+ 5028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:18.050795 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:18.051972 (+ 1177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:18.052314 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:18.052368 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 11:37:18.052491 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38971,"thread_start_us":38783,"threads_started":1}
I20260924 11:37:18.054126 21029 tablet_service.cc:1467] Processing CreateTablet for tablet 2d24d1dce0544a9893daeeb4dacab498 (DEFAULT_TABLE table=test-table [id=9246abc1026b435686fa8b8e73c918c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:37:18.054491 21163 tablet_service.cc:1467] Processing CreateTablet for tablet 2d24d1dce0544a9893daeeb4dacab498 (DEFAULT_TABLE table=test-table [id=9246abc1026b435686fa8b8e73c918c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:37:18.055711 21029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d24d1dce0544a9893daeeb4dacab498. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:37:18.055713 21163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d24d1dce0544a9893daeeb4dacab498. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:37:18.055450 21297 tablet_service.cc:1467] Processing CreateTablet for tablet 2d24d1dce0544a9893daeeb4dacab498 (DEFAULT_TABLE table=test-table [id=9246abc1026b435686fa8b8e73c918c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:37:18.056456 21297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d24d1dce0544a9893daeeb4dacab498. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:37:18.068034 21381 tablet_bootstrap.cc:492] T 2d24d1dce0544a9893daeeb4dacab498 P 1d695defd9734c80a12887ed84ede67c: Bootstrap starting.
I20260924 11:37:18.071784 21381 tablet_bootstrap.cc:654] T 2d24d1dce0544a9893daeeb4dacab498 P 1d695defd9734c80a12887ed84ede67c: Neither blocks nor log segments found. Creating new log.
I20260924 11:37:18.073091 21381 log.cc:824] T 2d24d1dce0544a9893daeeb4dacab498 P 1d695defd9734c80a12887ed84ede67c: Log is configured to *not* fsync() on all Append() calls
I20260924 11:37:18.076316 21381 tablet_bootstrap.cc:492] T 2d24d1dce0544a9893daeeb4dacab498 P 1d695defd9734c80a12887ed84ede67c: No bootstrap required, opened a new log
I20260924 11:37:18.076644 21381 ts_tablet_manager.cc:1397] T 2d24d1dce0544a9893daeeb4dacab498 P 1d695defd9734c80a12887ed84ede67c: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260924 11:37:18.080303 21383 tablet_bootstrap.cc:492] T 2d24d1dce0544a9893daeeb4dacab498 P 6cb7f6130bda4786bc162352b37ee6ca: Bootstrap starting.
I20260924 11:37:18.080617 21382 tablet_bootstrap.cc:492] T 2d24d1dce0544a9893daeeb4dacab498 P bbffa38f702a4e08b829ed069a47c103: Bootstrap starting.
I20260924 11:37:18.083877 21383 tablet_bootstrap.cc:654] T 2d24d1dce0544a9893daeeb4dacab498 P 6cb7f6130bda4786bc162352b37ee6ca: Neither blocks nor log segments found. Creating new log.
I20260924 11:37:18.084527 21382 tablet_bootstrap.cc:654] T 2d24d1dce0544a9893daeeb4dacab498 P bbffa38f702a4e08b829ed069a47c103: Neither blocks nor log segments found. Creating new log.
I20260924 11:37:18.084990 21383 log.cc:824] T 2d24d1dce0544a9893daeeb4dacab498 P 6cb7f6130bda4786bc162352b37ee6ca: Log is configured to *not* fsync() on all Append() calls
I20260924 11:37:18.085744 21382 log.cc:824] T 2d24d1dce0544a9893daeeb4dacab498 P bbffa38f702a4e08b829ed069a47c103: Log is configured to *not* fsync() on all Append() calls
I20260924 11:37:18.086292 21381 raft_consensus.cc:348] T 2d24d1dce0544a9893daeeb4dacab498 P 1d695defd9734c80a12887ed84ede67c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d695defd9734c80a12887ed84ede67c" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 44015 } } peers { permanent_uuid: "bbffa38f702a4e08b829ed069a47c103" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 40045 } } peers { permanent_uuid: "6cb7f6130bda4786bc162352b37ee6ca" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 43247 } }
I20260924 11:37:18.087044 21381 raft_consensus.cc:374] T 2d24d1dce0544a9893daeeb4dacab498 P 1d695defd9734c80a12887ed84ede67c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:37:18.087224 21381 raft_consensus.cc:729] T 2d24d1dce0544a9893daeeb4dacab498 P 1d695defd9734c80a12887ed84ede67c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d695defd9734c80a12887ed84ede67c, State: Initialized, Role: FOLLOWER
I20260924 11:37:18.088259 21383 tablet_bootstrap.cc:492] T 2d24d1dce0544a9893daeeb4dacab498 P 6cb7f6130bda4786bc162352b37ee6ca: No bootstrap required, opened a new log
I20260924 11:37:18.088555 21383 ts_tablet_manager.cc:1397] T 2d24d1dce0544a9893daeeb4dacab498 P 6cb7f6130bda4786bc162352b37ee6ca: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260924 11:37:18.088357 21381 consensus_queue.cc:260] T 2d24d1dce0544a9893daeeb4dacab498 P 1d695defd9734c80a12887ed84ede67c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d695defd9734c80a12887ed84ede67c" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 44015 } } peers { permanent_uuid: "bbffa38f702a4e08b829ed069a47c103" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 40045 } } peers { permanent_uuid: "6cb7f6130bda4786bc162352b37ee6ca" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 43247 } }
I20260924 11:37:18.089749 21362 heartbeater.cc:503] Master 127.6.212.62:37655 was elected leader, sending a full tablet report...
I20260924 11:37:18.092869 21381 ts_tablet_manager.cc:1428] T 2d24d1dce0544a9893daeeb4dacab498 P 1d695defd9734c80a12887ed84ede67c: Time spent starting tablet: real 0.016s user 0.009s sys 0.004s
I20260924 11:37:18.096066 21382 tablet_bootstrap.cc:492] T 2d24d1dce0544a9893daeeb4dacab498 P bbffa38f702a4e08b829ed069a47c103: No bootstrap required, opened a new log
I20260924 11:37:18.096355 21382 ts_tablet_manager.cc:1397] T 2d24d1dce0544a9893daeeb4dacab498 P bbffa38f702a4e08b829ed069a47c103: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.009s
I20260924 11:37:18.097740 21383 raft_consensus.cc:348] T 2d24d1dce0544a9893daeeb4dacab498 P 6cb7f6130bda4786bc162352b37ee6ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d695defd9734c80a12887ed84ede67c" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 44015 } } peers { permanent_uuid: "bbffa38f702a4e08b829ed069a47c103" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 40045 } } peers { permanent_uuid: "6cb7f6130bda4786bc162352b37ee6ca" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 43247 } }
I20260924 11:37:18.098161 21383 raft_consensus.cc:374] T 2d24d1dce0544a9893daeeb4dacab498 P 6cb7f6130bda4786bc162352b37ee6ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:37:18.098284 21383 raft_consensus.cc:729] T 2d24d1dce0544a9893daeeb4dacab498 P 6cb7f6130bda4786bc162352b37ee6ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cb7f6130bda4786bc162352b37ee6ca, State: Initialized, Role: FOLLOWER
I20260924 11:37:18.098905 21383 consensus_queue.cc:260] T 2d24d1dce0544a9893daeeb4dacab498 P 6cb7f6130bda4786bc162352b37ee6ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d695defd9734c80a12887ed84ede67c" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 44015 } } peers { permanent_uuid: "bbffa38f702a4e08b829ed069a47c103" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 40045 } } peers { permanent_uuid: "6cb7f6130bda4786bc162352b37ee6ca" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 43247 } }
I20260924 11:37:18.100004 21228 heartbeater.cc:503] Master 127.6.212.62:37655 was elected leader, sending a full tablet report...
I20260924 11:37:18.100625 21383 ts_tablet_manager.cc:1428] T 2d24d1dce0544a9893daeeb4dacab498 P 6cb7f6130bda4786bc162352b37ee6ca: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260924 11:37:18.105285 21382 raft_consensus.cc:348] T 2d24d1dce0544a9893daeeb4dacab498 P bbffa38f702a4e08b829ed069a47c103 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d695defd9734c80a12887ed84ede67c" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 44015 } } peers { permanent_uuid: "bbffa38f702a4e08b829ed069a47c103" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 40045 } } peers { permanent_uuid: "6cb7f6130bda4786bc162352b37ee6ca" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 43247 } }
I20260924 11:37:18.105679 21382 raft_consensus.cc:374] T 2d24d1dce0544a9893daeeb4dacab498 P bbffa38f702a4e08b829ed069a47c103 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:37:18.105782 21382 raft_consensus.cc:729] T 2d24d1dce0544a9893daeeb4dacab498 P bbffa38f702a4e08b829ed069a47c103 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbffa38f702a4e08b829ed069a47c103, State: Initialized, Role: FOLLOWER
I20260924 11:37:18.106330 21382 consensus_queue.cc:260] T 2d24d1dce0544a9893daeeb4dacab498 P bbffa38f702a4e08b829ed069a47c103 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d695defd9734c80a12887ed84ede67c" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 44015 } } peers { permanent_uuid: "bbffa38f702a4e08b829ed069a47c103" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 40045 } } peers { permanent_uuid: "6cb7f6130bda4786bc162352b37ee6ca" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 43247 } }
I20260924 11:37:18.108070 21382 ts_tablet_manager.cc:1428] T 2d24d1dce0544a9893daeeb4dacab498 P bbffa38f702a4e08b829ed069a47c103: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
W20260924 11:37:18.111636 21229 tablet.cc:2369] T 2d24d1dce0544a9893daeeb4dacab498 P 6cb7f6130bda4786bc162352b37ee6ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:37:18.176126 21363 tablet.cc:2369] T 2d24d1dce0544a9893daeeb4dacab498 P 1d695defd9734c80a12887ed84ede67c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:37:18.312448 21095 tablet.cc:2369] T 2d24d1dce0544a9893daeeb4dacab498 P bbffa38f702a4e08b829ed069a47c103: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:37:18.402519 21389 raft_consensus.cc:482] T 2d24d1dce0544a9893daeeb4dacab498 P bbffa38f702a4e08b829ed069a47c103 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:37:18.402833 21389 raft_consensus.cc:504] T 2d24d1dce0544a9893daeeb4dacab498 P bbffa38f702a4e08b829ed069a47c103 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d695defd9734c80a12887ed84ede67c" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 44015 } } peers { permanent_uuid: "bbffa38f702a4e08b829ed069a47c103" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 40045 } } peers { permanent_uuid: "6cb7f6130bda4786bc162352b37ee6ca" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 43247 } }
I20260924 11:37:18.404771 21389 leader_election.cc:288] T 2d24d1dce0544a9893daeeb4dacab498 P bbffa38f702a4e08b829ed069a47c103 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d695defd9734c80a12887ed84ede67c (127.6.212.3:44015), 6cb7f6130bda4786bc162352b37ee6ca (127.6.212.2:43247)
I20260924 11:37:18.411996 21376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:18.405330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:40877 (local address 127.6.212.3:44015)
0924 11:37:18.405595 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:18.405604 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:18.406297 (+ 693us) server_negotiation.cc:407] Connection header received
0924 11:37:18.406509 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:18.406514 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:18.406572 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:18.406674 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:37:18.407241 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:18.407984 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:18.410037 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:18.410952 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:18.411092 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:18.411203 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:18.411587 (+ 384us) server_negotiation.cc:299] Negotiation successful
0924 11:37:18.411716 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260924 11:37:18.411995 21379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:18.405391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:57973 (local address 127.6.212.2:43247)
0924 11:37:18.405649 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:18.405658 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:18.406353 (+ 695us) server_negotiation.cc:407] Connection header received
0924 11:37:18.406558 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:18.406562 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:18.406620 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:18.406715 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:37:18.407327 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:18.408063 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:18.410040 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:18.410998 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:18.411145 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:18.411252 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:18.411588 (+ 336us) server_negotiation.cc:299] Negotiation successful
0924 11:37:18.411723 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260924 11:37:18.412155 21390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:18.405027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.3:44015 (local address 127.6.212.1:40877)
0924 11:37:18.406174 (+ 1147us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:18.406191 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:37:18.406372 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:37:18.406689 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:37:18.406693 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:37:18.406716 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:37:18.407096 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:18.407109 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:18.408113 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:18.408119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:18.409906 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:18.409919 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:18.411168 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:18.411174 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:18.411342 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:18.411457 (+ 115us) client_negotiation.cc:769] Sending connection context
0924 11:37:18.411557 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 11:37:18.411715 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1016,"mutex_wait_us":113,"thread_start_us":809,"threads_started":1}
I20260924 11:37:18.412197 21391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:18.405105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.2:43247 (local address 127.6.212.1:57973)
0924 11:37:18.406241 (+ 1136us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:18.406256 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:37:18.406406 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:37:18.406728 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:37:18.406732 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:37:18.406753 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:37:18.407162 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:18.407171 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:18.408201 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:18.408206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:18.409882 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:18.409897 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:18.411219 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:18.411225 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:18.411418 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:18.411460 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 11:37:18.411557 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 11:37:18.411722 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1025,"mutex_wait_us":57,"thread_start_us":763,"threads_started":1}
I20260924 11:37:18.413317 21317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d24d1dce0544a9893daeeb4dacab498" candidate_uuid: "bbffa38f702a4e08b829ed069a47c103" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d695defd9734c80a12887ed84ede67c" is_pre_election: true
I20260924 11:37:18.413334 21183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d24d1dce0544a9893daeeb4dacab498" candidate_uuid: "bbffa38f702a4e08b829ed069a47c103" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cb7f6130bda4786bc162352b37ee6ca" is_pre_election: true
I20260924 11:37:18.413800 21317 raft_consensus.cc:2454] T 2d24d1dce0544a9893daeeb4dacab498 P 1d695defd9734c80a12887ed84ede67c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbffa38f702a4e08b829ed069a47c103 in term 0.
I20260924 11:37:18.413815 21183 raft_consensus.cc:2454] T 2d24d1dce0544a9893daeeb4dacab498 P 6cb7f6130bda4786bc162352b37ee6ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbffa38f702a4e08b829ed069a47c103 in term 0.
I20260924 11:37:18.414458 20983 leader_election.cc:302] T 2d24d1dce0544a9893daeeb4dacab498 P bbffa38f702a4e08b829ed069a47c103 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d695defd9734c80a12887ed84ede67c, bbffa38f702a4e08b829ed069a47c103; no voters:
I20260924 11:37:18.414832 21389 raft_consensus.cc:2781] T 2d24d1dce0544a9893daeeb4dacab498 P bbffa38f702a4e08b829ed069a47c103 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:37:18.414983 21389 raft_consensus.cc:482] T 2d24d1dce0544a9893daeeb4dacab498 P bbffa38f702a4e08b829ed069a47c103 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:37:18.415083 21389 raft_consensus.cc:3037] T 2d24d1dce0544a9893daeeb4dacab498 P bbffa38f702a4e08b829ed069a47c103 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:37:18.417276 21389 raft_consensus.cc:504] T 2d24d1dce0544a9893daeeb4dacab498 P bbffa38f702a4e08b829ed069a47c103 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d695defd9734c80a12887ed84ede67c" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 44015 } } peers { permanent_uuid: "bbffa38f702a4e08b829ed069a47c103" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 40045 } } peers { permanent_uuid: "6cb7f6130bda4786bc162352b37ee6ca" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 43247 } }
I20260924 11:37:18.417927 21389 leader_election.cc:288] T 2d24d1dce0544a9893daeeb4dacab498 P bbffa38f702a4e08b829ed069a47c103 [CANDIDATE]: Term 1 election: Requested vote from peers 1d695defd9734c80a12887ed84ede67c (127.6.212.3:44015), 6cb7f6130bda4786bc162352b37ee6ca (127.6.212.2:43247)
I20260924 11:37:18.418358 21317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d24d1dce0544a9893daeeb4dacab498" candidate_uuid: "bbffa38f702a4e08b829ed069a47c103" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d695defd9734c80a12887ed84ede67c"
I20260924 11:37:18.418396 21183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d24d1dce0544a9893daeeb4dacab498" candidate_uuid: "bbffa38f702a4e08b829ed069a47c103" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cb7f6130bda4786bc162352b37ee6ca"
I20260924 11:37:18.418558 21317 raft_consensus.cc:3037] T 2d24d1dce0544a9893daeeb4dacab498 P 1d695defd9734c80a12887ed84ede67c [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:37:18.418603 21183 raft_consensus.cc:3037] T 2d24d1dce0544a9893daeeb4dacab498 P 6cb7f6130bda4786bc162352b37ee6ca [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:37:18.420859 21317 raft_consensus.cc:2454] T 2d24d1dce0544a9893daeeb4dacab498 P 1d695defd9734c80a12887ed84ede67c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbffa38f702a4e08b829ed069a47c103 in term 1.
I20260924 11:37:18.420895 21183 raft_consensus.cc:2454] T 2d24d1dce0544a9893daeeb4dacab498 P 6cb7f6130bda4786bc162352b37ee6ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbffa38f702a4e08b829ed069a47c103 in term 1.
I20260924 11:37:18.421283 20983 leader_election.cc:302] T 2d24d1dce0544a9893daeeb4dacab498 P bbffa38f702a4e08b829ed069a47c103 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d695defd9734c80a12887ed84ede67c, bbffa38f702a4e08b829ed069a47c103; no voters:
I20260924 11:37:18.421553 21389 raft_consensus.cc:2781] T 2d24d1dce0544a9893daeeb4dacab498 P bbffa38f702a4e08b829ed069a47c103 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:37:18.422022 21389 raft_consensus.cc:686] T 2d24d1dce0544a9893daeeb4dacab498 P bbffa38f702a4e08b829ed069a47c103 [term 1 LEADER]: Becoming Leader. State: Replica: bbffa38f702a4e08b829ed069a47c103, State: Running, Role: LEADER
I20260924 11:37:18.422514 21389 consensus_queue.cc:237] T 2d24d1dce0544a9893daeeb4dacab498 P bbffa38f702a4e08b829ed069a47c103 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d695defd9734c80a12887ed84ede67c" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 44015 } } peers { permanent_uuid: "bbffa38f702a4e08b829ed069a47c103" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 40045 } } peers { permanent_uuid: "6cb7f6130bda4786bc162352b37ee6ca" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 43247 } }
I20260924 11:37:18.427735 20905 catalog_manager.cc:5495] T 2d24d1dce0544a9893daeeb4dacab498 P bbffa38f702a4e08b829ed069a47c103 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbffa38f702a4e08b829ed069a47c103 (127.6.212.1). New cstate: current_term: 1 leader_uuid: "bbffa38f702a4e08b829ed069a47c103" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d695defd9734c80a12887ed84ede67c" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 44015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbffa38f702a4e08b829ed069a47c103" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 40045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6cb7f6130bda4786bc162352b37ee6ca" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 43247 } health_report { overall_health: UNKNOWN } } }
I20260924 11:37:18.475577 21380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:18.470062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46646 (local address 127.6.212.1:40045)
0924 11:37:18.470254 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:18.470258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:18.470305 (+ 47us) server_negotiation.cc:407] Connection header received
0924 11:37:18.470490 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:18.470493 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:18.470589 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:18.470714 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:37:18.471145 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:18.472019 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:18.473436 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:18.473856 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:18.473951 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:18.474502 (+ 551us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:37:18.475013 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:37:18.475123 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:18.475216 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 11:37:18.475284 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 11:37:18.483606 21317 raft_consensus.cc:1261] T 2d24d1dce0544a9893daeeb4dacab498 P 1d695defd9734c80a12887ed84ede67c [term 1 FOLLOWER]: Refusing update from remote peer bbffa38f702a4e08b829ed069a47c103: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:37:18.483606 21183 raft_consensus.cc:1261] T 2d24d1dce0544a9893daeeb4dacab498 P 6cb7f6130bda4786bc162352b37ee6ca [term 1 FOLLOWER]: Refusing update from remote peer bbffa38f702a4e08b829ed069a47c103: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:37:18.484560 21392 consensus_queue.cc:1035] T 2d24d1dce0544a9893daeeb4dacab498 P bbffa38f702a4e08b829ed069a47c103 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d695defd9734c80a12887ed84ede67c" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 44015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:37:18.484907 21389 consensus_queue.cc:1035] T 2d24d1dce0544a9893daeeb4dacab498 P bbffa38f702a4e08b829ed069a47c103 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cb7f6130bda4786bc162352b37ee6ca" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 43247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:37:18.499202 21397 mvcc.cc:205] Tried to move back new op lower bound from 7332863338419314688 to 7332863338182889472. Current Snapshot: MvccSnapshot[applied={T|T < 7332863338419314688}]
I20260924 11:37:18.499603 21396 mvcc.cc:205] Tried to move back new op lower bound from 7332863338419314688 to 7332863338182889472. Current Snapshot: MvccSnapshot[applied={T|T < 7332863338419314688}]
I20260924 11:37:18.510915 20904 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:37:18.511264 20904 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:37:18.515070 20904 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd: Sending DeleteTablet for 3 replicas of tablet 2d24d1dce0544a9893daeeb4dacab498
I20260924 11:37:18.516355 21297 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d24d1dce0544a9893daeeb4dacab498 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:37:18 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:37384
I20260924 11:37:18.516551 21029 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d24d1dce0544a9893daeeb4dacab498 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:37:18 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:46636
I20260924 11:37:18.516642 21163 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d24d1dce0544a9893daeeb4dacab498 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:37:18 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:36896
I20260924 11:37:18.517588 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 20966
I20260924 11:37:18.529412 21407 tablet_replica.cc:330] T 2d24d1dce0544a9893daeeb4dacab498 P 6cb7f6130bda4786bc162352b37ee6ca: stopping tablet replica
I20260924 11:37:18.530283 21407 raft_consensus.cc:2229] T 2d24d1dce0544a9893daeeb4dacab498 P 6cb7f6130bda4786bc162352b37ee6ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:37:18.531158 21407 raft_consensus.cc:2258] T 2d24d1dce0544a9893daeeb4dacab498 P 6cb7f6130bda4786bc162352b37ee6ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:37:18.534386 21407 ts_tablet_manager.cc:1905] T 2d24d1dce0544a9893daeeb4dacab498 P 6cb7f6130bda4786bc162352b37ee6ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:37:18.542338 21406 tablet_replica.cc:330] T 2d24d1dce0544a9893daeeb4dacab498 P bbffa38f702a4e08b829ed069a47c103: stopping tablet replica
I20260924 11:37:18.542969 21406 raft_consensus.cc:2229] T 2d24d1dce0544a9893daeeb4dacab498 P bbffa38f702a4e08b829ed069a47c103 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:37:18.543905 21406 raft_consensus.cc:2258] T 2d24d1dce0544a9893daeeb4dacab498 P bbffa38f702a4e08b829ed069a47c103 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:37:18.544571 21405 tablet_replica.cc:330] T 2d24d1dce0544a9893daeeb4dacab498 P 1d695defd9734c80a12887ed84ede67c: stopping tablet replica
I20260924 11:37:18.545365 21405 raft_consensus.cc:2229] T 2d24d1dce0544a9893daeeb4dacab498 P 1d695defd9734c80a12887ed84ede67c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:37:18.545914 21407 ts_tablet_manager.cc:1918] T 2d24d1dce0544a9893daeeb4dacab498 P 6cb7f6130bda4786bc162352b37ee6ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:37:18.546202 21407 log.cc:1196] T 2d24d1dce0544a9893daeeb4dacab498 P 6cb7f6130bda4786bc162352b37ee6ca: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-1/wal/wals/2d24d1dce0544a9893daeeb4dacab498
I20260924 11:37:18.546469 21405 raft_consensus.cc:2258] T 2d24d1dce0544a9893daeeb4dacab498 P 1d695defd9734c80a12887ed84ede67c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:37:18.546767 21407 ts_tablet_manager.cc:1939] T 2d24d1dce0544a9893daeeb4dacab498 P 6cb7f6130bda4786bc162352b37ee6ca: Deleting consensus metadata
I20260924 11:37:18.548600 21406 ts_tablet_manager.cc:1905] T 2d24d1dce0544a9893daeeb4dacab498 P bbffa38f702a4e08b829ed069a47c103: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:37:18.548600 21405 ts_tablet_manager.cc:1905] T 2d24d1dce0544a9893daeeb4dacab498 P 1d695defd9734c80a12887ed84ede67c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:37:18.548817 20891 catalog_manager.cc:4841] TS 6cb7f6130bda4786bc162352b37ee6ca (127.6.212.2:43247): tablet 2d24d1dce0544a9893daeeb4dacab498 (table test-table [id=9246abc1026b435686fa8b8e73c918c4]) successfully deleted
Sep 24 11:37:18 dist-test-slave-cq7c krb5kdc[20860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.1@KRBTEST.COM
I20260924 11:37:18.556180 21405 ts_tablet_manager.cc:1918] T 2d24d1dce0544a9893daeeb4dacab498 P 1d695defd9734c80a12887ed84ede67c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:37:18.556782 21405 log.cc:1196] T 2d24d1dce0544a9893daeeb4dacab498 P 1d695defd9734c80a12887ed84ede67c: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-2/wal/wals/2d24d1dce0544a9893daeeb4dacab498
I20260924 11:37:18.557436 21405 ts_tablet_manager.cc:1939] T 2d24d1dce0544a9893daeeb4dacab498 P 1d695defd9734c80a12887ed84ede67c: Deleting consensus metadata
I20260924 11:37:18.557847 21406 ts_tablet_manager.cc:1918] T 2d24d1dce0544a9893daeeb4dacab498 P bbffa38f702a4e08b829ed069a47c103: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:37:18.558086 21406 log.cc:1196] T 2d24d1dce0544a9893daeeb4dacab498 P bbffa38f702a4e08b829ed069a47c103: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-0/wal/wals/2d24d1dce0544a9893daeeb4dacab498
I20260924 11:37:18.558719 21406 ts_tablet_manager.cc:1939] T 2d24d1dce0544a9893daeeb4dacab498 P bbffa38f702a4e08b829ed069a47c103: Deleting consensus metadata
I20260924 11:37:18.559562 20890 catalog_manager.cc:4841] TS 1d695defd9734c80a12887ed84ede67c (127.6.212.3:44015): tablet 2d24d1dce0544a9893daeeb4dacab498 (table test-table [id=9246abc1026b435686fa8b8e73c918c4]) successfully deleted
I20260924 11:37:18.560665 20892 catalog_manager.cc:4841] TS bbffa38f702a4e08b829ed069a47c103 (127.6.212.1:40045): tablet 2d24d1dce0544a9893daeeb4dacab498 (table test-table [id=9246abc1026b435686fa8b8e73c918c4]) successfully deleted
I20260924 11:37:18.564253 21380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:18.521459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46652 (local address 127.6.212.1:40045)
0924 11:37:18.521704 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:18.521714 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:18.541423 (+ 19709us) server_negotiation.cc:407] Connection header received
0924 11:37:18.541645 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:18.541654 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:18.541773 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:18.541934 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:18.545818 (+ 3884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:18.547656 (+ 1838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:18.549843 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:18.550647 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:18.550772 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:18.556930 (+ 6158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:18.556948 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:18.556971 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:18.557035 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:18.561172 (+ 4137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:18.561967 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:18.561974 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:18.561977 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:18.562119 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:18.562545 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:18.562551 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:18.562553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:18.562903 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:18.563208 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:18.563597 (+ 389us) server_negotiation.cc:299] Negotiation successful
0924 11:37:18.563865 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":55}
I20260924 11:37:18.564930 21089 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:37:18.667248 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 20966
I20260924 11:37:18.686650 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 21100
Sep 24 11:37:18 dist-test-slave-cq7c krb5kdc[20860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.2@KRBTEST.COM
I20260924 11:37:18.706908 21379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:18.687333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36908 (local address 127.6.212.2:43247)
0924 11:37:18.687530 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:18.687534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:18.687578 (+ 44us) server_negotiation.cc:407] Connection header received
0924 11:37:18.687729 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:18.687733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:18.687798 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:18.687893 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:18.689403 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:18.690163 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:18.691727 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:18.692194 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:18.692297 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:18.701251 (+ 8954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:18.701268 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:18.701286 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:18.701333 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:18.704387 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:18.705075 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:18.705081 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:18.705084 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:18.705214 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:18.705541 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:18.705544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:18.705546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:18.705913 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:18.706131 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:18.706383 (+ 252us) server_negotiation.cc:299] Negotiation successful
0924 11:37:18.706548 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 11:37:18.707669 21223 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:37:18.798597 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 21100
I20260924 11:37:18.816479 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 21234
Sep 24 11:37:18 dist-test-slave-cq7c krb5kdc[20860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.3@KRBTEST.COM
I20260924 11:37:18.833413 21376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:18.817275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37388 (local address 127.6.212.3:44015)
0924 11:37:18.817458 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:18.817463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:18.817482 (+ 19us) server_negotiation.cc:407] Connection header received
0924 11:37:18.817604 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:18.817609 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:18.817685 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:18.817771 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:18.819445 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:18.820249 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:18.821708 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:18.822203 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:18.822313 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:18.827197 (+ 4884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:18.827218 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:18.827259 (+ 41us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:18.827295 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:18.830298 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:18.831111 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:18.831120 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:18.831122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:18.831277 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:18.831844 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:18.831848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:18.831850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:18.832269 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:18.832492 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:18.832824 (+ 332us) server_negotiation.cc:299] Negotiation successful
0924 11:37:18.833086 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 11:37:18.834051 21357 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:37:18.923875 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 21234
I20260924 11:37:18.942049 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 20873
I20260924 11:37:18.943217 20935 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:37:19.058537 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 20873
I20260924 11:37:19.081549 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:37655
--webserver_interface=127.6.212.62
--webserver_port=37835
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:36593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:37655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:37:19.431895 21413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:37:19.432219 21413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:37:19.432312 21413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:37:19.439661 21413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:37:19.439782 21413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:37:19.439827 21413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:37:19.439883 21413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:37:19.439929 21413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:37:19.449908 21413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:36593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:37655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:37655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.212.62
--webserver_port=37835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:37:19.451251 21413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:37:19.452625 21413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:37:19.480358 21419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:37:19.480396 21420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:37:19.485898 21413 server_base.cc:1034] running on GCE node
W20260924 11:37:19.487820 21422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:37:19.488902 21413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:37:19.490684 21413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:37:19.491925 21413 hybrid_clock.cc:648] HybridClock initialized: now 1790249839491858 us; error 63 us; skew 500 ppm
I20260924 11:37:19.492400 21413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:37:19 dist-test-slave-cq7c krb5kdc[20860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249839, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:37:19.499883 21413 init.cc:377] Logged in from keytab as kudu/127.6.212.62@KRBTEST.COM (short username kudu)
I20260924 11:37:19.501957 21413 webserver.cc:460] Webserver started at http://127.6.212.62:37835/ using document root <none> and password file <none>
I20260924 11:37:19.502835 21413 fs_manager.cc:359] Metadata directory not provided
I20260924 11:37:19.503014 21413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:37:19.509470 21413 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 11:37:19.512722 21429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:19.514285 21413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:37:19.514493 21413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "c9d70886d7b64358958769d14824cdbd"
format_stamp: "Formatted at 2026-09-24 11:37:16 on dist-test-slave-cq7c"
server_key: "c8bfd4e83afd1a56d29bbe80e0d24a98"
server_key_iv: "60f6236b101981a12fbd31c87a8258e3"
server_key_version: "encryptionkey@0"
I20260924 11:37:19.515734 21413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:37:19.543710 21413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:37:19.548480 21413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:37:19.548770 21413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:37:19.568423 21413 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.62:37655
I20260924 11:37:19.568464 21480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.62:37655 every 8 connection(s)
I20260924 11:37:19.570127 21413 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:37:19.576684 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 21413
I20260924 11:37:19.577741 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.1:40045
--local_ip_for_outbound_sockets=127.6.212.1
--tserver_master_addrs=127.6.212.62:37655
--webserver_port=38555
--webserver_interface=127.6.212.1
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab
--principal=kudu/127.6.212.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:36593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:37:19.578797 21481 sys_catalog.cc:263] Verifying existing consensus state
I20260924 11:37:19.584628 21481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd: Bootstrap starting.
I20260924 11:37:19.605960 21481 log.cc:824] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd: Log is configured to *not* fsync() on all Append() calls
I20260924 11:37:19.618338 21481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 11:37:19.618897 21481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd: Bootstrap complete.
I20260924 11:37:19.629166 21481 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d70886d7b64358958769d14824cdbd" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 37655 } }
I20260924 11:37:19.630054 21481 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9d70886d7b64358958769d14824cdbd, State: Initialized, Role: FOLLOWER
I20260924 11:37:19.630707 21481 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d70886d7b64358958769d14824cdbd" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 37655 } }
I20260924 11:37:19.630950 21481 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:37:19.631067 21481 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:37:19.631248 21481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [term 1 FOLLOWER]: Advancing to term 2
I20260924 11:37:19.635898 21481 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d70886d7b64358958769d14824cdbd" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 37655 } }
I20260924 11:37:19.636497 21481 leader_election.cc:302] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9d70886d7b64358958769d14824cdbd; no voters:
I20260924 11:37:19.637207 21481 leader_election.cc:288] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 11:37:19.637373 21485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [term 2 FOLLOWER]: Leader election won for term 2
I20260924 11:37:19.638726 21485 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [term 2 LEADER]: Becoming Leader. State: Replica: c9d70886d7b64358958769d14824cdbd, State: Running, Role: LEADER
I20260924 11:37:19.639379 21485 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d70886d7b64358958769d14824cdbd" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 37655 } }
I20260924 11:37:19.639787 21481 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:37:19.641983 21487 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9d70886d7b64358958769d14824cdbd. Latest consensus state: current_term: 2 leader_uuid: "c9d70886d7b64358958769d14824cdbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d70886d7b64358958769d14824cdbd" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 37655 } } }
I20260924 11:37:19.642365 21487 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [sys.catalog]: This master's current role is: LEADER
I20260924 11:37:19.642107 21486 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c9d70886d7b64358958769d14824cdbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d70886d7b64358958769d14824cdbd" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 37655 } } }
I20260924 11:37:19.642542 21486 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [sys.catalog]: This master's current role is: LEADER
I20260924 11:37:19.649184 21492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:37:19.658552 21492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:37:19.660135 21492 catalog_manager.cc:1259] Loaded cluster ID: dd49dfdc53b54a94bb35fde2787bbb5c
I20260924 11:37:19.660303 21492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:37:19.664615 21492 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 11:37:19.682211 21492 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 11:37:19.691594 21492 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:37:19.693466 21492 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd: Loaded TSK: 0
I20260924 11:37:19.695459 21492 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 11:37:19.943162 21483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:37:19.943521 21483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:37:19.943686 21483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:37:19.951390 21483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:37:19.951514 21483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:37:19.951687 21483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.1
I20260924 11:37:19.961889 21483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:36593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab
--principal=kudu/127.6.212.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.1:40045
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.212.1
--webserver_port=38555
--tserver_master_addrs=127.6.212.62:37655
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.1
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:37:19.963398 21483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:37:19.964757 21483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:37:19.980482 21509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:37:19.980835 21511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:37:19.980494 21508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:37:19.982286 21483 server_base.cc:1034] running on GCE node
I20260924 11:37:19.983129 21483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:37:19.984493 21483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:37:19.985760 21483 hybrid_clock.cc:648] HybridClock initialized: now 1790249839985627 us; error 119 us; skew 500 ppm
I20260924 11:37:19.986315 21483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:37:19 dist-test-slave-cq7c krb5kdc[20860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249839, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:37:19.990736 21483 init.cc:377] Logged in from keytab as kudu/127.6.212.1@KRBTEST.COM (short username kudu)
I20260924 11:37:19.992795 21483 webserver.cc:460] Webserver started at http://127.6.212.1:38555/ using document root <none> and password file <none>
I20260924 11:37:19.993563 21483 fs_manager.cc:359] Metadata directory not provided
I20260924 11:37:19.993718 21483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:37:19.999856 21483 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260924 11:37:20.002827 21518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:20.004943 21483 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 11:37:20.005159 21483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-0/wal
uuid: "bbffa38f702a4e08b829ed069a47c103"
format_stamp: "Formatted at 2026-09-24 11:37:16 on dist-test-slave-cq7c"
server_key: "2a7fcf6e99679b71e950e59b74a52b03"
server_key_iv: "d6364fb283b253faa7a850615bff5b73"
server_key_version: "encryptionkey@0"
I20260924 11:37:20.006238 21483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:37:20.032105 21483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:37:20.036860 21483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:37:20.037146 21483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:37:20.038208 21483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:37:20.039660 21483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:37:20.039772 21483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:20.039908 21483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:37:20.039994 21483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:20.074165 21483 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.1:40045
I20260924 11:37:20.074260 21630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.1:40045 every 8 connection(s)
I20260924 11:37:20.075762 21483 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
I20260924 11:37:20.078163 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 21483
I20260924 11:37:20.078955 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.2:43247
--local_ip_for_outbound_sockets=127.6.212.2
--tserver_master_addrs=127.6.212.62:37655
--webserver_port=35125
--webserver_interface=127.6.212.2
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab
--principal=kudu/127.6.212.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:36593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 11:37:20 dist-test-slave-cq7c krb5kdc[20860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249839, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:37:20.103255 21633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:20.079253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:50815 (local address 127.6.212.62:37655)
0924 11:37:20.080025 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:20.080044 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:20.080711 (+ 667us) server_negotiation.cc:407] Connection header received
0924 11:37:20.082979 (+ 2268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:20.083004 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:20.083552 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:20.084009 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:20.085538 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:20.087337 (+ 1799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:20.090013 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:20.090726 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:20.090970 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:20.095426 (+ 4456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:20.095436 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:20.095453 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:20.095499 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:20.098611 (+ 3112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:20.099410 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:20.099417 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:20.099422 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:20.099588 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:20.100022 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:20.100028 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:20.100032 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:20.100513 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:20.100719 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:20.101520 (+ 801us) server_negotiation.cc:299] Negotiation successful
0924 11:37:20.101907 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":305,"threads_started":1}
I20260924 11:37:20.103255 21634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:20.079475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.1:50815)
0924 11:37:20.080478 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:20.080542 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 11:37:20.082312 (+ 1770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:37:20.084177 (+ 1865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:37:20.084185 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:37:20.084721 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:37:20.085271 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:20.085290 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:20.087592 (+ 2302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:20.087601 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:20.089846 (+ 2245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:20.089865 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:20.091056 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:20.091066 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:20.091418 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:20.092214 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:37:20.092249 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:37:20.095066 (+ 2817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:37:20.098766 (+ 3700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:20.098776 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:20.098792 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:20.099232 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:20.099710 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:20.099715 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:20.099719 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:20.099910 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:20.100660 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:37:20.100669 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:37:20.101024 (+ 355us) client_negotiation.cc:769] Sending connection context
0924 11:37:20.101316 (+ 292us) client_negotiation.cc:240] Negotiation successful
0924 11:37:20.101894 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"thread_start_us":459,"threads_started":1}
I20260924 11:37:20.106431 21631 heartbeater.cc:347] Connected to a master server at 127.6.212.62:37655
I20260924 11:37:20.106747 21631 heartbeater.cc:464] Registering TS with master...
I20260924 11:37:20.107510 21631 heartbeater.cc:511] Master 127.6.212.62:37655 requested a full tablet report, sending...
I20260924 11:37:20.109858 21446 ts_manager.cc:194] Registered new tserver with Master: bbffa38f702a4e08b829ed069a47c103 (127.6.212.1:40045)
I20260924 11:37:20.111549 21446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.1@KRBTEST.COM'} at 127.6.212.1:50815
W20260924 11:37:20.431689 21635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:37:20.432152 21635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:37:20.432325 21635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:37:20.444365 21635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:37:20.444536 21635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:37:20.444743 21635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.2
I20260924 11:37:20.455199 21635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:36593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab
--principal=kudu/127.6.212.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.2:43247
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.212.2
--webserver_port=35125
--tserver_master_addrs=127.6.212.62:37655
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.2
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:37:20.456614 21635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:37:20.457984 21635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:37:20.472564 21640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:37:20.472536 21643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:37:20.472780 21641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:37:20.474073 21635 server_base.cc:1034] running on GCE node
I20260924 11:37:20.474700 21635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:37:20.475752 21635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:37:20.476974 21635 hybrid_clock.cc:648] HybridClock initialized: now 1790249840476927 us; error 39 us; skew 500 ppm
I20260924 11:37:20.477334 21635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:37:20 dist-test-slave-cq7c krb5kdc[20860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249840, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:37:20.480731 21635 init.cc:377] Logged in from keytab as kudu/127.6.212.2@KRBTEST.COM (short username kudu)
I20260924 11:37:20.482451 21635 webserver.cc:460] Webserver started at http://127.6.212.2:35125/ using document root <none> and password file <none>
I20260924 11:37:20.483191 21635 fs_manager.cc:359] Metadata directory not provided
I20260924 11:37:20.483352 21635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:37:20.488956 21635 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:37:20.491859 21650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:20.493455 21635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 11:37:20.493654 21635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-1/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-1/wal
uuid: "6cb7f6130bda4786bc162352b37ee6ca"
format_stamp: "Formatted at 2026-09-24 11:37:17 on dist-test-slave-cq7c"
server_key: "e1642fcd032ee8456f94393532c660ee"
server_key_iv: "6bc1321ed194db0a5bc1e8395b4f0829"
server_key_version: "encryptionkey@0"
I20260924 11:37:20.494704 21635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:37:20.512081 21635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:37:20.516357 21635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:37:20.516647 21635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:37:20.517686 21635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:37:20.519085 21635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:37:20.519197 21635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:20.519330 21635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:37:20.519446 21635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260924 11:37:20.552999 21635 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.2:43247
I20260924 11:37:20.553153 21762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.2:43247 every 8 connection(s)
I20260924 11:37:20.554595 21635 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
I20260924 11:37:20.556643 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 21635
I20260924 11:37:20.557284 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.3:44015
--local_ip_for_outbound_sockets=127.6.212.3
--tserver_master_addrs=127.6.212.62:37655
--webserver_port=43683
--webserver_interface=127.6.212.3
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab
--principal=kudu/127.6.212.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:36593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 11:37:20 dist-test-slave-cq7c krb5kdc[20860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249840, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.2@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:37:20.581755 21633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:20.557563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:41297 (local address 127.6.212.62:37655)
0924 11:37:20.557783 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:20.557790 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:20.559269 (+ 1479us) server_negotiation.cc:407] Connection header received
0924 11:37:20.562406 (+ 3137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:20.562415 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:20.562513 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:20.562615 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:20.564359 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:20.565088 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:20.567553 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:20.568286 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:20.568431 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:20.573358 (+ 4927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:20.573368 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:20.573376 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:20.573428 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:20.577466 (+ 4038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:20.578565 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:20.578574 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:20.578580 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:20.578766 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:20.579395 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:20.579403 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:20.579409 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:20.579855 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:20.580032 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:20.580969 (+ 937us) server_negotiation.cc:299] Negotiation successful
0924 11:37:20.581276 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 11:37:20.582901 21765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:20.558165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.2:41297)
0924 11:37:20.559065 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:20.559116 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 11:37:20.561983 (+ 2867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:37:20.562899 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:37:20.562908 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:37:20.563537 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:37:20.564164 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:20.564182 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:20.565261 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:20.565266 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:20.567352 (+ 2086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:20.567400 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:20.568547 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:20.568557 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:20.568900 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:20.569772 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:37:20.569807 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:37:20.573058 (+ 3251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:37:20.577683 (+ 4625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:20.577694 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:20.577715 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:20.578394 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:20.578896 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:20.578901 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:20.578904 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:20.579169 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:20.580011 (+ 842us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:37:20.580019 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:37:20.580464 (+ 445us) client_negotiation.cc:769] Sending connection context
0924 11:37:20.580785 (+ 321us) client_negotiation.cc:240] Negotiation successful
0924 11:37:20.581380 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":321,"threads_started":1}
I20260924 11:37:20.584628 21763 heartbeater.cc:347] Connected to a master server at 127.6.212.62:37655
I20260924 11:37:20.585000 21763 heartbeater.cc:464] Registering TS with master...
I20260924 11:37:20.586012 21763 heartbeater.cc:511] Master 127.6.212.62:37655 requested a full tablet report, sending...
I20260924 11:37:20.587702 21446 ts_manager.cc:194] Registered new tserver with Master: 6cb7f6130bda4786bc162352b37ee6ca (127.6.212.2:43247)
I20260924 11:37:20.588562 21446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.2@KRBTEST.COM'} at 127.6.212.2:41297
W20260924 11:37:20.901115 21766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:37:20.901389 21766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:37:20.901542 21766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:37:20.908777 21766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:37:20.908883 21766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:37:20.909070 21766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.3
I20260924 11:37:20.918994 21766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:36593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab
--principal=kudu/127.6.212.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.3:44015
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.212.3
--webserver_port=43683
--tserver_master_addrs=127.6.212.62:37655
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.3
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:37:20.920521 21766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:37:20.921866 21766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:37:20.934895 21772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:37:20.934908 21771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:37:20.935505 21774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:37:20.935959 21766 server_base.cc:1034] running on GCE node
I20260924 11:37:20.936571 21766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:37:20.937557 21766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:37:20.938792 21766 hybrid_clock.cc:648] HybridClock initialized: now 1790249840938722 us; error 59 us; skew 500 ppm
I20260924 11:37:20.939117 21766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:37:20 dist-test-slave-cq7c krb5kdc[20860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249840, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:37:20.942631 21766 init.cc:377] Logged in from keytab as kudu/127.6.212.3@KRBTEST.COM (short username kudu)
I20260924 11:37:20.944437 21766 webserver.cc:460] Webserver started at http://127.6.212.3:43683/ using document root <none> and password file <none>
I20260924 11:37:20.945241 21766 fs_manager.cc:359] Metadata directory not provided
I20260924 11:37:20.945416 21766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:37:20.951893 21766 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 11:37:20.954847 21781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:20.956399 21766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:37:20.956588 21766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-2/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-2/wal
uuid: "1d695defd9734c80a12887ed84ede67c"
format_stamp: "Formatted at 2026-09-24 11:37:17 on dist-test-slave-cq7c"
server_key: "70770c8ecb3d8f8f6046c7648a791921"
server_key_iv: "dac51743039c1ff5ea727efd5dcac7ee"
server_key_version: "encryptionkey@0"
I20260924 11:37:20.957679 21766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:37:20.975901 21766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:37:20.981278 21766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:37:20.981570 21766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:37:20.982611 21766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:37:20.984011 21766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:37:20.984133 21766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:20.984259 21766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:37:20.984354 21766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:21.020176 21766 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.3:44015
I20260924 11:37:21.020237 21893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.3:44015 every 8 connection(s)
I20260924 11:37:21.021765 21766 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
I20260924 11:37:21.026701 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 21766
Sep 24 11:37:21 dist-test-slave-cq7c krb5kdc[20860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249840, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.3@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:37:21.042759 21897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:21.027591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33574 (local address 127.6.212.62:37655)
0924 11:37:21.028106 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:21.028112 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:21.028428 (+ 316us) server_negotiation.cc:407] Connection header received
0924 11:37:21.028612 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:21.028617 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:21.028683 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:21.028793 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:21.030239 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:21.031077 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:21.032453 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:21.033037 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:21.033131 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:21.036313 (+ 3182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:21.036320 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:21.036325 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:21.036366 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:21.039775 (+ 3409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:21.040669 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:21.040677 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:21.040682 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:21.040823 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:21.041226 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:21.041231 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:21.041249 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:21.041557 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:21.041842 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:21.042141 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 11:37:21.042384 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":385,"thread_start_us":255,"threads_started":1}
I20260924 11:37:21.043934 21633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:21.024508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:59639 (local address 127.6.212.62:37655)
0924 11:37:21.024721 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:21.024727 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:21.025973 (+ 1246us) server_negotiation.cc:407] Connection header received
0924 11:37:21.028280 (+ 2307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:21.028286 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:21.028377 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:21.028484 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:21.030119 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:21.030926 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:21.032665 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:21.033277 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:21.033407 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:21.037443 (+ 4036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:21.037448 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:21.037451 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:21.037494 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:21.040550 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:21.041427 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:21.041436 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:21.041438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:21.041566 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:21.042013 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:21.042018 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:21.042021 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:21.042273 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:21.042427 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:21.043298 (+ 871us) server_negotiation.cc:299] Negotiation successful
0924 11:37:21.043531 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":70}
I20260924 11:37:21.045073 21896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:21.025086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.3:59639)
0924 11:37:21.025804 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:21.025847 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 11:37:21.027905 (+ 2058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:37:21.028793 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:37:21.028805 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:37:21.029380 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:37:21.029946 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:21.029965 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:21.031086 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:21.031091 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:21.032527 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:21.032541 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:21.033499 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:21.033504 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:21.033722 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:21.034367 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:37:21.034389 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:37:21.037156 (+ 2767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:37:21.040723 (+ 3567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:21.040734 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:21.040756 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:21.041298 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:21.041678 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:21.041683 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:21.041687 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:21.041886 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:21.042400 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:37:21.042408 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:37:21.042859 (+ 451us) client_negotiation.cc:769] Sending connection context
0924 11:37:21.043211 (+ 352us) client_negotiation.cc:240] Negotiation successful
0924 11:37:21.043791 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":285,"threads_started":1}
I20260924 11:37:21.046476 21894 heartbeater.cc:347] Connected to a master server at 127.6.212.62:37655
I20260924 11:37:21.046789 21894 heartbeater.cc:464] Registering TS with master...
I20260924 11:37:21.047566 21894 heartbeater.cc:511] Master 127.6.212.62:37655 requested a full tablet report, sending...
I20260924 11:37:21.048996 21446 ts_manager.cc:194] Registered new tserver with Master: 1d695defd9734c80a12887ed84ede67c (127.6.212.3:44015)
I20260924 11:37:21.049860 21446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.3@KRBTEST.COM'} at 127.6.212.3:59639
I20260924 11:37:21.058310 6992 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:37:21.065557 21633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:21.059232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33576 (local address 127.6.212.62:37655)
0924 11:37:21.059439 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:21.059443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:21.059916 (+ 473us) server_negotiation.cc:407] Connection header received
0924 11:37:21.060152 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:21.060156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:21.060286 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:21.060436 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:37:21.060885 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:21.061679 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:21.063321 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:21.063810 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:21.063957 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:21.064476 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:37:21.064988 (+ 512us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:37:21.065155 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:21.065206 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 11:37:21.065291 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 11:37:21.066641 21446 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:33576:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:37:21.074651 21446 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:37:21.100630 21904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:21.088263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.2:43247 (local address 127.0.0.1:36916)
0924 11:37:21.090261 (+ 1998us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:21.090282 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 11:37:21.090426 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:37:21.092023 (+ 1597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:37:21.092035 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:37:21.092079 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:37:21.092786 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:21.092812 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:21.095297 (+ 2485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:21.095303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:21.098001 (+ 2698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:21.098018 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:21.099626 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:21.099634 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:21.099828 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:21.099883 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 11:37:21.100035 (+ 152us) client_negotiation.cc:240] Negotiation successful
0924 11:37:21.100241 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1828,"thread_start_us":453,"threads_started":1}
I20260924 11:37:21.100627 21907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:21.088636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36916 (local address 127.6.212.2:43247)
0924 11:37:21.091189 (+ 2553us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:21.091202 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:21.091239 (+ 37us) server_negotiation.cc:407] Connection header received
0924 11:37:21.091419 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:21.091432 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:21.091752 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:21.092476 (+ 724us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:37:21.093723 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:21.095164 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:21.098189 (+ 3025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:21.099439 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:21.099576 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:21.099804 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:21.100113 (+ 309us) server_negotiation.cc:299] Negotiation successful
0924 11:37:21.100296 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2355,"thread_start_us":2271,"threads_started":1}
I20260924 11:37:21.102152 21905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:21.088068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:40045 (local address 127.0.0.1:46662)
0924 11:37:21.089262 (+ 1194us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:21.089337 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0924 11:37:21.089547 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:37:21.093801 (+ 4254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:37:21.093805 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:37:21.093830 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:37:21.094277 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:21.094294 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:21.096607 (+ 2313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:21.096612 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:21.099153 (+ 2541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:21.099166 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:21.101250 (+ 2084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:21.101257 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:21.101450 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:21.101485 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 11:37:21.101553 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 11:37:21.101715 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"mutex_wait_us":66,"thread_start_us":1836,"threads_started":1}
I20260924 11:37:21.102572 21908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:21.090558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.3:44015 (local address 127.0.0.1:37396)
0924 11:37:21.093374 (+ 2816us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:21.093392 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:37:21.093508 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:37:21.094186 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:37:21.094191 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:37:21.094223 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:37:21.094643 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:21.094653 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:21.096644 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:21.096648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:21.099151 (+ 2503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:21.099166 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:21.101710 (+ 2544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:21.101715 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:21.101921 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:21.101952 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 11:37:21.102049 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 11:37:21.102207 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2680,"mutex_wait_us":165,"thread_start_us":2532,"threads_started":1}
I20260924 11:37:21.103300 21909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:21.090952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37396 (local address 127.6.212.3:44015)
0924 11:37:21.092114 (+ 1162us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:21.092125 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:21.093505 (+ 1380us) server_negotiation.cc:407] Connection header received
0924 11:37:21.093717 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:21.093726 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:21.093958 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:21.094214 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:37:21.094862 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:21.096488 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:21.099426 (+ 2938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:21.100953 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:21.101075 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:21.101307 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:21.102804 (+ 1497us) server_negotiation.cc:299] Negotiation successful
0924 11:37:21.102988 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":935,"thread_start_us":751,"threads_started":1}
I20260924 11:37:21.103649 21906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:21.088374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46662 (local address 127.6.212.1:40045)
0924 11:37:21.093195 (+ 4821us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:21.093202 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:21.093232 (+ 30us) server_negotiation.cc:407] Connection header received
0924 11:37:21.093337 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:21.093347 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:21.093604 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:21.094511 (+ 907us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:37:21.094683 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:21.096439 (+ 1756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:21.099439 (+ 3000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:21.100978 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:21.102607 (+ 1629us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:21.103059 (+ 452us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:21.103128 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 11:37:21.103288 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4646,"thread_start_us":4499,"threads_started":1}
I20260924 11:37:21.105163 21698 tablet_service.cc:1467] Processing CreateTablet for tablet 8649defef81d4cf5bfbd406fe54378e9 (DEFAULT_TABLE table=test-table [id=f870d744ca1a418287f29f5006d424d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:37:21.106719 21698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8649defef81d4cf5bfbd406fe54378e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:37:21.107226 21566 tablet_service.cc:1467] Processing CreateTablet for tablet 8649defef81d4cf5bfbd406fe54378e9 (DEFAULT_TABLE table=test-table [id=f870d744ca1a418287f29f5006d424d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:37:21.108785 21566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8649defef81d4cf5bfbd406fe54378e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:37:21.108906 21829 tablet_service.cc:1467] Processing CreateTablet for tablet 8649defef81d4cf5bfbd406fe54378e9 (DEFAULT_TABLE table=test-table [id=f870d744ca1a418287f29f5006d424d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:37:21.110080 21829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8649defef81d4cf5bfbd406fe54378e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:37:21.115031 21631 heartbeater.cc:503] Master 127.6.212.62:37655 was elected leader, sending a full tablet report...
I20260924 11:37:21.123026 21910 tablet_bootstrap.cc:492] T 8649defef81d4cf5bfbd406fe54378e9 P 1d695defd9734c80a12887ed84ede67c: Bootstrap starting.
I20260924 11:37:21.125183 21911 tablet_bootstrap.cc:492] T 8649defef81d4cf5bfbd406fe54378e9 P 6cb7f6130bda4786bc162352b37ee6ca: Bootstrap starting.
I20260924 11:37:21.127018 21912 tablet_bootstrap.cc:492] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103: Bootstrap starting.
I20260924 11:37:21.127040 21910 tablet_bootstrap.cc:654] T 8649defef81d4cf5bfbd406fe54378e9 P 1d695defd9734c80a12887ed84ede67c: Neither blocks nor log segments found. Creating new log.
I20260924 11:37:21.128397 21910 log.cc:824] T 8649defef81d4cf5bfbd406fe54378e9 P 1d695defd9734c80a12887ed84ede67c: Log is configured to *not* fsync() on all Append() calls
I20260924 11:37:21.128914 21911 tablet_bootstrap.cc:654] T 8649defef81d4cf5bfbd406fe54378e9 P 6cb7f6130bda4786bc162352b37ee6ca: Neither blocks nor log segments found. Creating new log.
I20260924 11:37:21.130359 21912 tablet_bootstrap.cc:654] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103: Neither blocks nor log segments found. Creating new log.
I20260924 11:37:21.130465 21911 log.cc:824] T 8649defef81d4cf5bfbd406fe54378e9 P 6cb7f6130bda4786bc162352b37ee6ca: Log is configured to *not* fsync() on all Append() calls
I20260924 11:37:21.131748 21912 log.cc:824] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103: Log is configured to *not* fsync() on all Append() calls
I20260924 11:37:21.133369 21910 tablet_bootstrap.cc:492] T 8649defef81d4cf5bfbd406fe54378e9 P 1d695defd9734c80a12887ed84ede67c: No bootstrap required, opened a new log
I20260924 11:37:21.133461 21911 tablet_bootstrap.cc:492] T 8649defef81d4cf5bfbd406fe54378e9 P 6cb7f6130bda4786bc162352b37ee6ca: No bootstrap required, opened a new log
I20260924 11:37:21.133674 21910 ts_tablet_manager.cc:1397] T 8649defef81d4cf5bfbd406fe54378e9 P 1d695defd9734c80a12887ed84ede67c: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260924 11:37:21.133724 21911 ts_tablet_manager.cc:1397] T 8649defef81d4cf5bfbd406fe54378e9 P 6cb7f6130bda4786bc162352b37ee6ca: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260924 11:37:21.134326 21912 tablet_bootstrap.cc:492] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103: No bootstrap required, opened a new log
I20260924 11:37:21.134596 21912 ts_tablet_manager.cc:1397] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260924 11:37:21.142467 21912 raft_consensus.cc:348] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbffa38f702a4e08b829ed069a47c103" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 40045 } } peers { permanent_uuid: "6cb7f6130bda4786bc162352b37ee6ca" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 43247 } } peers { permanent_uuid: "1d695defd9734c80a12887ed84ede67c" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 44015 } }
I20260924 11:37:21.142910 21912 raft_consensus.cc:374] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:37:21.143033 21912 raft_consensus.cc:729] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbffa38f702a4e08b829ed069a47c103, State: Initialized, Role: FOLLOWER
I20260924 11:37:21.143721 21912 consensus_queue.cc:260] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbffa38f702a4e08b829ed069a47c103" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 40045 } } peers { permanent_uuid: "6cb7f6130bda4786bc162352b37ee6ca" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 43247 } } peers { permanent_uuid: "1d695defd9734c80a12887ed84ede67c" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 44015 } }
I20260924 11:37:21.144982 21910 raft_consensus.cc:348] T 8649defef81d4cf5bfbd406fe54378e9 P 1d695defd9734c80a12887ed84ede67c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbffa38f702a4e08b829ed069a47c103" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 40045 } } peers { permanent_uuid: "6cb7f6130bda4786bc162352b37ee6ca" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 43247 } } peers { permanent_uuid: "1d695defd9734c80a12887ed84ede67c" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 44015 } }
I20260924 11:37:21.144981 21911 raft_consensus.cc:348] T 8649defef81d4cf5bfbd406fe54378e9 P 6cb7f6130bda4786bc162352b37ee6ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbffa38f702a4e08b829ed069a47c103" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 40045 } } peers { permanent_uuid: "6cb7f6130bda4786bc162352b37ee6ca" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 43247 } } peers { permanent_uuid: "1d695defd9734c80a12887ed84ede67c" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 44015 } }
I20260924 11:37:21.145431 21911 raft_consensus.cc:374] T 8649defef81d4cf5bfbd406fe54378e9 P 6cb7f6130bda4786bc162352b37ee6ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:37:21.145521 21910 raft_consensus.cc:374] T 8649defef81d4cf5bfbd406fe54378e9 P 1d695defd9734c80a12887ed84ede67c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:37:21.145587 21911 raft_consensus.cc:729] T 8649defef81d4cf5bfbd406fe54378e9 P 6cb7f6130bda4786bc162352b37ee6ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cb7f6130bda4786bc162352b37ee6ca, State: Initialized, Role: FOLLOWER
I20260924 11:37:21.145632 21912 ts_tablet_manager.cc:1428] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260924 11:37:21.145668 21910 raft_consensus.cc:729] T 8649defef81d4cf5bfbd406fe54378e9 P 1d695defd9734c80a12887ed84ede67c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d695defd9734c80a12887ed84ede67c, State: Initialized, Role: FOLLOWER
I20260924 11:37:21.146307 21911 consensus_queue.cc:260] T 8649defef81d4cf5bfbd406fe54378e9 P 6cb7f6130bda4786bc162352b37ee6ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbffa38f702a4e08b829ed069a47c103" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 40045 } } peers { permanent_uuid: "6cb7f6130bda4786bc162352b37ee6ca" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 43247 } } peers { permanent_uuid: "1d695defd9734c80a12887ed84ede67c" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 44015 } }
I20260924 11:37:21.146307 21910 consensus_queue.cc:260] T 8649defef81d4cf5bfbd406fe54378e9 P 1d695defd9734c80a12887ed84ede67c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbffa38f702a4e08b829ed069a47c103" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 40045 } } peers { permanent_uuid: "6cb7f6130bda4786bc162352b37ee6ca" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 43247 } } peers { permanent_uuid: "1d695defd9734c80a12887ed84ede67c" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 44015 } }
I20260924 11:37:21.147980 21763 heartbeater.cc:503] Master 127.6.212.62:37655 was elected leader, sending a full tablet report...
I20260924 11:37:21.148342 21910 ts_tablet_manager.cc:1428] T 8649defef81d4cf5bfbd406fe54378e9 P 1d695defd9734c80a12887ed84ede67c: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
I20260924 11:37:21.148512 21911 ts_tablet_manager.cc:1428] T 8649defef81d4cf5bfbd406fe54378e9 P 6cb7f6130bda4786bc162352b37ee6ca: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260924 11:37:21.148874 21894 heartbeater.cc:503] Master 127.6.212.62:37655 was elected leader, sending a full tablet report...
I20260924 11:37:21.236901 21918 raft_consensus.cc:482] T 8649defef81d4cf5bfbd406fe54378e9 P 6cb7f6130bda4786bc162352b37ee6ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:37:21.237249 21918 raft_consensus.cc:504] T 8649defef81d4cf5bfbd406fe54378e9 P 6cb7f6130bda4786bc162352b37ee6ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbffa38f702a4e08b829ed069a47c103" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 40045 } } peers { permanent_uuid: "6cb7f6130bda4786bc162352b37ee6ca" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 43247 } } peers { permanent_uuid: "1d695defd9734c80a12887ed84ede67c" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 44015 } }
I20260924 11:37:21.239078 21918 leader_election.cc:288] T 8649defef81d4cf5bfbd406fe54378e9 P 6cb7f6130bda4786bc162352b37ee6ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbffa38f702a4e08b829ed069a47c103 (127.6.212.1:40045), 1d695defd9734c80a12887ed84ede67c (127.6.212.3:44015)
I20260924 11:37:21.246348 21906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:21.239757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:59617 (local address 127.6.212.1:40045)
0924 11:37:21.240013 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:21.240022 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:21.240813 (+ 791us) server_negotiation.cc:407] Connection header received
0924 11:37:21.241055 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:21.241061 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:21.241124 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:21.241218 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:37:21.241733 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:21.242465 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:21.244636 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:21.245408 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:21.245562 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:21.245679 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:21.245963 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 11:37:21.246095 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 11:37:21.246451 21919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:21.239594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:40045 (local address 127.6.212.2:59617)
0924 11:37:21.240682 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:21.240698 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:37:21.240899 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:37:21.241234 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:37:21.241239 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:37:21.241273 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:37:21.241594 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:21.241603 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:21.242590 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:21.242598 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:21.244491 (+ 1893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:21.244509 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:21.245638 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:21.245645 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:21.245797 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:21.245822 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 11:37:21.245927 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 11:37:21.246069 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":966,"thread_start_us":691,"threads_started":1}
I20260924 11:37:21.246706 21909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:21.239753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:38531 (local address 127.6.212.3:44015)
0924 11:37:21.240045 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:21.240054 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:21.241255 (+ 1201us) server_negotiation.cc:407] Connection header received
0924 11:37:21.241543 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:21.241549 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:21.241612 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:21.241703 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:37:21.242233 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:21.242946 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:21.245022 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:21.245802 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:21.245951 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:21.246063 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:21.246333 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 11:37:21.246473 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260924 11:37:21.247056 21920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:21.240137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.3:44015 (local address 127.6.212.2:38531)
0924 11:37:21.241139 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:21.241157 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:37:21.241396 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:37:21.241718 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:37:21.241723 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:37:21.241746 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:37:21.242089 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:21.242096 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:21.243083 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:21.243092 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:21.244869 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:21.244880 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:21.246022 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:21.246028 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:21.246184 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:21.246210 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 11:37:21.246307 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 11:37:21.246435 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":882,"spinlock_wait_cycles":39936,"thread_start_us":625,"threads_started":1}
I20260924 11:37:21.247669 21586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8649defef81d4cf5bfbd406fe54378e9" candidate_uuid: "6cb7f6130bda4786bc162352b37ee6ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbffa38f702a4e08b829ed069a47c103" is_pre_election: true
I20260924 11:37:21.248052 21849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8649defef81d4cf5bfbd406fe54378e9" candidate_uuid: "6cb7f6130bda4786bc162352b37ee6ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d695defd9734c80a12887ed84ede67c" is_pre_election: true
I20260924 11:37:21.248203 21586 raft_consensus.cc:2454] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cb7f6130bda4786bc162352b37ee6ca in term 0.
I20260924 11:37:21.248529 21849 raft_consensus.cc:2454] T 8649defef81d4cf5bfbd406fe54378e9 P 1d695defd9734c80a12887ed84ede67c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cb7f6130bda4786bc162352b37ee6ca in term 0.
I20260924 11:37:21.248906 21654 leader_election.cc:302] T 8649defef81d4cf5bfbd406fe54378e9 P 6cb7f6130bda4786bc162352b37ee6ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cb7f6130bda4786bc162352b37ee6ca, bbffa38f702a4e08b829ed069a47c103; no voters:
I20260924 11:37:21.249377 21918 raft_consensus.cc:2781] T 8649defef81d4cf5bfbd406fe54378e9 P 6cb7f6130bda4786bc162352b37ee6ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:37:21.249538 21918 raft_consensus.cc:482] T 8649defef81d4cf5bfbd406fe54378e9 P 6cb7f6130bda4786bc162352b37ee6ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:37:21.249653 21918 raft_consensus.cc:3037] T 8649defef81d4cf5bfbd406fe54378e9 P 6cb7f6130bda4786bc162352b37ee6ca [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:37:21.252038 21918 raft_consensus.cc:504] T 8649defef81d4cf5bfbd406fe54378e9 P 6cb7f6130bda4786bc162352b37ee6ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbffa38f702a4e08b829ed069a47c103" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 40045 } } peers { permanent_uuid: "6cb7f6130bda4786bc162352b37ee6ca" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 43247 } } peers { permanent_uuid: "1d695defd9734c80a12887ed84ede67c" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 44015 } }
I20260924 11:37:21.252687 21918 leader_election.cc:288] T 8649defef81d4cf5bfbd406fe54378e9 P 6cb7f6130bda4786bc162352b37ee6ca [CANDIDATE]: Term 1 election: Requested vote from peers bbffa38f702a4e08b829ed069a47c103 (127.6.212.1:40045), 1d695defd9734c80a12887ed84ede67c (127.6.212.3:44015)
I20260924 11:37:21.253099 21586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8649defef81d4cf5bfbd406fe54378e9" candidate_uuid: "6cb7f6130bda4786bc162352b37ee6ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbffa38f702a4e08b829ed069a47c103"
I20260924 11:37:21.253232 21849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8649defef81d4cf5bfbd406fe54378e9" candidate_uuid: "6cb7f6130bda4786bc162352b37ee6ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d695defd9734c80a12887ed84ede67c"
I20260924 11:37:21.253311 21586 raft_consensus.cc:3037] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:37:21.253458 21849 raft_consensus.cc:3037] T 8649defef81d4cf5bfbd406fe54378e9 P 1d695defd9734c80a12887ed84ede67c [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:37:21.255720 21586 raft_consensus.cc:2454] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cb7f6130bda4786bc162352b37ee6ca in term 1.
I20260924 11:37:21.255792 21849 raft_consensus.cc:2454] T 8649defef81d4cf5bfbd406fe54378e9 P 1d695defd9734c80a12887ed84ede67c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cb7f6130bda4786bc162352b37ee6ca in term 1.
I20260924 11:37:21.256150 21654 leader_election.cc:302] T 8649defef81d4cf5bfbd406fe54378e9 P 6cb7f6130bda4786bc162352b37ee6ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cb7f6130bda4786bc162352b37ee6ca, bbffa38f702a4e08b829ed069a47c103; no voters:
I20260924 11:37:21.256448 21918 raft_consensus.cc:2781] T 8649defef81d4cf5bfbd406fe54378e9 P 6cb7f6130bda4786bc162352b37ee6ca [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:37:21.256893 21918 raft_consensus.cc:686] T 8649defef81d4cf5bfbd406fe54378e9 P 6cb7f6130bda4786bc162352b37ee6ca [term 1 LEADER]: Becoming Leader. State: Replica: 6cb7f6130bda4786bc162352b37ee6ca, State: Running, Role: LEADER
I20260924 11:37:21.257408 21918 consensus_queue.cc:237] T 8649defef81d4cf5bfbd406fe54378e9 P 6cb7f6130bda4786bc162352b37ee6ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbffa38f702a4e08b829ed069a47c103" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 40045 } } peers { permanent_uuid: "6cb7f6130bda4786bc162352b37ee6ca" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 43247 } } peers { permanent_uuid: "1d695defd9734c80a12887ed84ede67c" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 44015 } }
I20260924 11:37:21.262056 21444 catalog_manager.cc:5495] T 8649defef81d4cf5bfbd406fe54378e9 P 6cb7f6130bda4786bc162352b37ee6ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 6cb7f6130bda4786bc162352b37ee6ca (127.6.212.2). New cstate: current_term: 1 leader_uuid: "6cb7f6130bda4786bc162352b37ee6ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbffa38f702a4e08b829ed069a47c103" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 40045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6cb7f6130bda4786bc162352b37ee6ca" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 43247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d695defd9734c80a12887ed84ede67c" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 44015 } health_report { overall_health: UNKNOWN } } }
W20260924 11:37:21.273428 21895 tablet.cc:2369] T 8649defef81d4cf5bfbd406fe54378e9 P 1d695defd9734c80a12887ed84ede67c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:37:21.294394 21907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:21.289177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36922 (local address 127.6.212.2:43247)
0924 11:37:21.289370 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:21.289375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:21.289393 (+ 18us) server_negotiation.cc:407] Connection header received
0924 11:37:21.289584 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:21.289588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:21.289668 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:21.289796 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:37:21.290114 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:21.290838 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:21.292317 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:21.292714 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:21.292808 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:21.293295 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:37:21.293811 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:37:21.293969 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:21.294008 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 11:37:21.294073 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 11:37:21.302898 21849 raft_consensus.cc:1261] T 8649defef81d4cf5bfbd406fe54378e9 P 1d695defd9734c80a12887ed84ede67c [term 1 FOLLOWER]: Refusing update from remote peer 6cb7f6130bda4786bc162352b37ee6ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:37:21.302898 21586 raft_consensus.cc:1261] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103 [term 1 FOLLOWER]: Refusing update from remote peer 6cb7f6130bda4786bc162352b37ee6ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:37:21.303929 21921 consensus_queue.cc:1035] T 8649defef81d4cf5bfbd406fe54378e9 P 6cb7f6130bda4786bc162352b37ee6ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d695defd9734c80a12887ed84ede67c" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 44015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:37:21.304454 21918 consensus_queue.cc:1035] T 8649defef81d4cf5bfbd406fe54378e9 P 6cb7f6130bda4786bc162352b37ee6ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbffa38f702a4e08b829ed069a47c103" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 40045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 11:37:21.306563 21764 tablet.cc:2369] T 8649defef81d4cf5bfbd406fe54378e9 P 6cb7f6130bda4786bc162352b37ee6ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:37:21.331547 21445 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:33576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:37:21.331835 21445 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:33576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:37:21.335619 21445 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd: Sending DeleteTablet for 3 replicas of tablet 8649defef81d4cf5bfbd406fe54378e9
I20260924 11:37:21.336898 21829 tablet_service.cc:1514] Processing DeleteTablet for tablet 8649defef81d4cf5bfbd406fe54378e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:37:21 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:37396
I20260924 11:37:21.336905 21698 tablet_service.cc:1514] Processing DeleteTablet for tablet 8649defef81d4cf5bfbd406fe54378e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:37:21 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:36916
I20260924 11:37:21.336901 21566 tablet_service.cc:1514] Processing DeleteTablet for tablet 8649defef81d4cf5bfbd406fe54378e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:37:21 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:46662
I20260924 11:37:21.341339 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 21483
I20260924 11:37:21.349411 21936 tablet_replica.cc:330] T 8649defef81d4cf5bfbd406fe54378e9 P 1d695defd9734c80a12887ed84ede67c: stopping tablet replica
I20260924 11:37:21.349617 21935 tablet_replica.cc:330] T 8649defef81d4cf5bfbd406fe54378e9 P 6cb7f6130bda4786bc162352b37ee6ca: stopping tablet replica
I20260924 11:37:21.350126 21935 raft_consensus.cc:2229] T 8649defef81d4cf5bfbd406fe54378e9 P 6cb7f6130bda4786bc162352b37ee6ca [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:37:21.350116 21936 raft_consensus.cc:2229] T 8649defef81d4cf5bfbd406fe54378e9 P 1d695defd9734c80a12887ed84ede67c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:37:21.350703 21936 raft_consensus.cc:2258] T 8649defef81d4cf5bfbd406fe54378e9 P 1d695defd9734c80a12887ed84ede67c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:37:21.350855 21935 raft_consensus.cc:2258] T 8649defef81d4cf5bfbd406fe54378e9 P 6cb7f6130bda4786bc162352b37ee6ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:37:21.351621 21934 tablet_replica.cc:330] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103: stopping tablet replica
I20260924 11:37:21.352108 21934 raft_consensus.cc:2229] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:37:21.352669 21934 raft_consensus.cc:2258] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:37:21.353487 21936 ts_tablet_manager.cc:1905] T 8649defef81d4cf5bfbd406fe54378e9 P 1d695defd9734c80a12887ed84ede67c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:37:21.354770 21935 ts_tablet_manager.cc:1905] T 8649defef81d4cf5bfbd406fe54378e9 P 6cb7f6130bda4786bc162352b37ee6ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:37:21.360921 21936 ts_tablet_manager.cc:1918] T 8649defef81d4cf5bfbd406fe54378e9 P 1d695defd9734c80a12887ed84ede67c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:37:21.361232 21936 log.cc:1196] T 8649defef81d4cf5bfbd406fe54378e9 P 1d695defd9734c80a12887ed84ede67c: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-2/wal/wals/8649defef81d4cf5bfbd406fe54378e9
I20260924 11:37:21.361994 21936 ts_tablet_manager.cc:1939] T 8649defef81d4cf5bfbd406fe54378e9 P 1d695defd9734c80a12887ed84ede67c: Deleting consensus metadata
I20260924 11:37:21.363010 21935 ts_tablet_manager.cc:1918] T 8649defef81d4cf5bfbd406fe54378e9 P 6cb7f6130bda4786bc162352b37ee6ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:37:21.363286 21935 log.cc:1196] T 8649defef81d4cf5bfbd406fe54378e9 P 6cb7f6130bda4786bc162352b37ee6ca: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-1/wal/wals/8649defef81d4cf5bfbd406fe54378e9
I20260924 11:37:21.364106 21935 ts_tablet_manager.cc:1939] T 8649defef81d4cf5bfbd406fe54378e9 P 6cb7f6130bda4786bc162352b37ee6ca: Deleting consensus metadata
I20260924 11:37:21.364192 21431 catalog_manager.cc:4841] TS 1d695defd9734c80a12887ed84ede67c (127.6.212.3:44015): tablet 8649defef81d4cf5bfbd406fe54378e9 (table test-table [id=f870d744ca1a418287f29f5006d424d5]) successfully deleted
I20260924 11:37:21.363677 21906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:21.342941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46676 (local address 127.6.212.1:40045)
0924 11:37:21.343123 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:21.343129 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:21.343148 (+ 19us) server_negotiation.cc:407] Connection header received
0924 11:37:21.343208 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:21.343213 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:21.343312 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:21.343502 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:21.345532 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:21.346689 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:21.348699 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:21.349399 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:21.349543 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:21.354187 (+ 4644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:21.354203 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:21.354221 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:21.354269 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:21.358776 (+ 4507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:21.359969 (+ 1193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:21.359979 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:21.359983 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:21.360160 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:21.360687 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:21.360692 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:21.360696 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:21.361231 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:21.361549 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:21.362848 (+ 1299us) server_negotiation.cc:299] Negotiation successful
0924 11:37:21.363253 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 11:37:21.365067 21934 ts_tablet_manager.cc:1905] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:37:21.366137 21626 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:37:21.366314 21432 catalog_manager.cc:4841] TS 6cb7f6130bda4786bc162352b37ee6ca (127.6.212.2:43247): tablet 8649defef81d4cf5bfbd406fe54378e9 (table test-table [id=f870d744ca1a418287f29f5006d424d5]) successfully deleted
I20260924 11:37:21.478912 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 21483
I20260924 11:37:21.497211 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 21635
W20260924 11:37:21.497220 21433 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.212.1:40045 (error 108)
I20260924 11:37:21.499567 21908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:21.498989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:40045 (local address 127.0.0.1:46680)
0924 11:37:21.499208 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:21.499347 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.1:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260924 11:37:21.499914 21433 catalog_manager.cc:4584] TS bbffa38f702a4e08b829ed069a47c103 (127.6.212.1:40045): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8649defef81d4cf5bfbd406fe54378e9: Network error: Client connection negotiation failed: client connection to 127.6.212.1:40045: connect: Connection refused (error 111)
I20260924 11:37:21.514532 21907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:21.498481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36926 (local address 127.6.212.2:43247)
0924 11:37:21.498709 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:21.498716 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:21.498871 (+ 155us) server_negotiation.cc:407] Connection header received
0924 11:37:21.499086 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:21.499091 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:21.499177 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:21.499287 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:21.501179 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:21.502029 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:21.504173 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:21.504675 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:21.504773 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:21.508187 (+ 3414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:21.508202 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:21.508222 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:21.508261 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:21.511625 (+ 3364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:21.512498 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:21.512505 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:21.512508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:21.512654 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:21.513055 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:21.513058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:21.513060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:21.513383 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:21.513620 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:21.513997 (+ 377us) server_negotiation.cc:299] Negotiation successful
0924 11:37:21.514163 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 11:37:21.515211 21758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:37:21.553773 21908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:21.553163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:40045 (local address 127.0.0.1:46682)
0924 11:37:21.553446 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:21.553576 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.1:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260924 11:37:21.554847 21908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:21.554456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:40045 (local address 127.0.0.1:46694)
0924 11:37:21.554606 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:21.554696 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.1:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 11:37:21.615566 21908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:21.614964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:40045 (local address 127.0.0.1:46702)
0924 11:37:21.615206 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:21.615337 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.1:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260924 11:37:21.616598 21908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:21.616273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:40045 (local address 127.0.0.1:46704)
0924 11:37:21.616410 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:21.616482 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.1:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 11:37:21.631946 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 21635
I20260924 11:37:21.651222 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 21766
I20260924 11:37:21.666177 21909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:21.652035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37408 (local address 127.6.212.3:44015)
0924 11:37:21.652231 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:21.652235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:21.652255 (+ 20us) server_negotiation.cc:407] Connection header received
0924 11:37:21.652430 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:21.652434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:21.652501 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:21.652598 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:21.653942 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:21.654663 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:21.656175 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:21.656589 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:21.656704 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:21.659943 (+ 3239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:21.659958 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:21.659974 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:21.660019 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:21.663339 (+ 3320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:21.664268 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:21.664276 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:21.664278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:21.664444 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:21.664904 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:21.664908 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:21.664910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:21.665250 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:21.665509 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:21.665730 (+ 221us) server_negotiation.cc:299] Negotiation successful
0924 11:37:21.665906 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 11:37:21.666819 21889 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:37:21.697381 21908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:21.696832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:40045 (local address 127.0.0.1:46718)
0924 11:37:21.697086 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:21.697185 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.1:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260924 11:37:21.698549 21908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:21.698103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:40045 (local address 127.0.0.1:46730)
0924 11:37:21.698348 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:21.698426 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.1:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260924 11:37:21.761018 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 21766
I20260924 11:37:21.778087 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 21413
I20260924 11:37:21.779294 21476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:37:21.895293 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 21413
I20260924 11:37:21.917759 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:37655
--webserver_interface=127.6.212.62
--webserver_port=37835
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:36593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:37655
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:37:22.264323 21942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:37:22.264626 21942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:37:22.264801 21942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:37:22.272203 21942 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:37:22.272315 21942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:37:22.272409 21942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:37:22.272506 21942 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:37:22.272581 21942 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:37:22.282963 21942 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:36593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:37655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:37655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.212.62
--webserver_port=37835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:37:22.284441 21942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:37:22.285897 21942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:37:22.298482 21950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:37:22.298564 21948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:37:22.298554 21947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:37:22.301079 21942 server_base.cc:1034] running on GCE node
I20260924 11:37:22.301981 21942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:37:22.303536 21942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:37:22.304762 21942 hybrid_clock.cc:648] HybridClock initialized: now 1790249842304689 us; error 93 us; skew 500 ppm
I20260924 11:37:22.305192 21942 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:37:22 dist-test-slave-cq7c krb5kdc[20860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249842, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:37:22.308439 21942 init.cc:377] Logged in from keytab as kudu/127.6.212.62@KRBTEST.COM (short username kudu)
I20260924 11:37:22.310081 21942 webserver.cc:460] Webserver started at http://127.6.212.62:37835/ using document root <none> and password file <none>
I20260924 11:37:22.310801 21942 fs_manager.cc:359] Metadata directory not provided
I20260924 11:37:22.310943 21942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:37:22.316480 21942 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 11:37:22.319461 21957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:22.320978 21942 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 11:37:22.321156 21942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "c9d70886d7b64358958769d14824cdbd"
format_stamp: "Formatted at 2026-09-24 11:37:16 on dist-test-slave-cq7c"
server_key: "c8bfd4e83afd1a56d29bbe80e0d24a98"
server_key_iv: "60f6236b101981a12fbd31c87a8258e3"
server_key_version: "encryptionkey@0"
I20260924 11:37:22.322170 21942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:37:22.342427 21942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:37:22.346835 21942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:37:22.347131 21942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:37:22.366564 21942 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.62:37655
I20260924 11:37:22.366571 22008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.62:37655 every 8 connection(s)
I20260924 11:37:22.368238 21942 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:37:22.373042 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 21942
I20260924 11:37:22.374017 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.1:40045
--local_ip_for_outbound_sockets=127.6.212.1
--tserver_master_addrs=127.6.212.62:37655
--webserver_port=38555
--webserver_interface=127.6.212.1
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab
--principal=kudu/127.6.212.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:36593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:37:22.376192 22009 sys_catalog.cc:263] Verifying existing consensus state
I20260924 11:37:22.381652 22009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd: Bootstrap starting.
I20260924 11:37:22.403035 22009 log.cc:824] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd: Log is configured to *not* fsync() on all Append() calls
I20260924 11:37:22.419258 22009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 11:37:22.419850 22009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd: Bootstrap complete.
I20260924 11:37:22.429976 22009 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d70886d7b64358958769d14824cdbd" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 37655 } }
I20260924 11:37:22.430887 22009 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9d70886d7b64358958769d14824cdbd, State: Initialized, Role: FOLLOWER
I20260924 11:37:22.431568 22009 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d70886d7b64358958769d14824cdbd" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 37655 } }
I20260924 11:37:22.431809 22009 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:37:22.431950 22009 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:37:22.432124 22009 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [term 2 FOLLOWER]: Advancing to term 3
I20260924 11:37:22.438248 22009 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d70886d7b64358958769d14824cdbd" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 37655 } }
I20260924 11:37:22.438803 22009 leader_election.cc:302] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9d70886d7b64358958769d14824cdbd; no voters:
I20260924 11:37:22.439564 22009 leader_election.cc:288] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 11:37:22.439703 22013 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [term 3 FOLLOWER]: Leader election won for term 3
I20260924 11:37:22.441005 22013 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [term 3 LEADER]: Becoming Leader. State: Replica: c9d70886d7b64358958769d14824cdbd, State: Running, Role: LEADER
I20260924 11:37:22.441689 22013 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d70886d7b64358958769d14824cdbd" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 37655 } }
I20260924 11:37:22.441901 22009 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:37:22.444370 22015 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9d70886d7b64358958769d14824cdbd. Latest consensus state: current_term: 3 leader_uuid: "c9d70886d7b64358958769d14824cdbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d70886d7b64358958769d14824cdbd" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 37655 } } }
I20260924 11:37:22.444752 22015 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [sys.catalog]: This master's current role is: LEADER
I20260924 11:37:22.445300 22014 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c9d70886d7b64358958769d14824cdbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9d70886d7b64358958769d14824cdbd" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 37655 } } }
I20260924 11:37:22.445617 22014 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9d70886d7b64358958769d14824cdbd [sys.catalog]: This master's current role is: LEADER
I20260924 11:37:22.456810 22020 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:37:22.466782 22020 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:37:22.468372 22020 catalog_manager.cc:1259] Loaded cluster ID: dd49dfdc53b54a94bb35fde2787bbb5c
I20260924 11:37:22.468546 22020 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 11:37:22.470698 22020 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f76dade49ad google::LogMessage::Fail() at ??:0
@ 0x7f76dade788a google::LogMessage::SendToLog() at ??:0
@ 0x7f76dade44de google::LogMessage::Flush() at ??:0
@ 0x7f76dade5a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f76f0817512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f76f0814976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
I20260924 11:37:22.495229 22031 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f76f089c610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f76f089c439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f76dda30345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f76dda3391c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f76dda33799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f76dda0eeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f76e0ce86db start_thread at ??:0
@ 0x7f76d9c7e61f clone at ??:0
W20260924 11:37:22.765929 22011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:37:22.766314 22011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:37:22.766491 22011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:37:22.774286 22011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:37:22.774480 22011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:37:22.774667 22011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.1
I20260924 11:37:22.786098 22011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:36593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab
--principal=kudu/127.6.212.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.1:40045
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.212.1
--webserver_port=38555
--tserver_master_addrs=127.6.212.62:37655
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.1
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:37:22.787611 22011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:37:22.789017 22011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:37:22.803041 22050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:37:22.803102 22053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:37:22.803124 22051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:37:22.804198 22011 server_base.cc:1034] running on GCE node
I20260924 11:37:22.804853 22011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:37:22.805871 22011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:37:22.807103 22011 hybrid_clock.cc:648] HybridClock initialized: now 1790249842807046 us; error 51 us; skew 500 ppm
I20260924 11:37:22.807492 22011 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:37:22 dist-test-slave-cq7c krb5kdc[20860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249842, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:37:22.810689 22011 init.cc:377] Logged in from keytab as kudu/127.6.212.1@KRBTEST.COM (short username kudu)
I20260924 11:37:22.812163 22011 webserver.cc:460] Webserver started at http://127.6.212.1:38555/ using document root <none> and password file <none>
I20260924 11:37:22.812855 22011 fs_manager.cc:359] Metadata directory not provided
I20260924 11:37:22.812952 22011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:37:22.818293 22011 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 11:37:22.821506 22060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:22.822999 22011 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 11:37:22.823136 22011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-0/wal
uuid: "bbffa38f702a4e08b829ed069a47c103"
format_stamp: "Formatted at 2026-09-24 11:37:16 on dist-test-slave-cq7c"
server_key: "2a7fcf6e99679b71e950e59b74a52b03"
server_key_iv: "d6364fb283b253faa7a850615bff5b73"
server_key_version: "encryptionkey@0"
I20260924 11:37:22.824246 22011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:37:22.843720 22011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:37:22.847926 22011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:37:22.848152 22011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:37:22.849146 22011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:37:22.850975 22067 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260924 11:37:22.856781 22011 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 11:37:22.856914 22011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260924 11:37:22.857008 22011 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 11:37:22.860497 22011 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 11:37:22.860628 22011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.003s sys 0.000s
I20260924 11:37:22.861033 22067 tablet_bootstrap.cc:492] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103: Bootstrap starting.
I20260924 11:37:22.870560 22067 log.cc:824] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103: Log is configured to *not* fsync() on all Append() calls
I20260924 11:37:22.884776 22067 tablet_bootstrap.cc:492] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 11:37:22.885767 22067 tablet_bootstrap.cc:492] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103: Bootstrap complete.
I20260924 11:37:22.886368 22067 ts_tablet_manager.cc:1397] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103: Time spent bootstrapping tablet: real 0.026s user 0.015s sys 0.008s
I20260924 11:37:22.897603 22067 raft_consensus.cc:348] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbffa38f702a4e08b829ed069a47c103" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 40045 } } peers { permanent_uuid: "6cb7f6130bda4786bc162352b37ee6ca" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 43247 } } peers { permanent_uuid: "1d695defd9734c80a12887ed84ede67c" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 44015 } }
I20260924 11:37:22.899035 22067 raft_consensus.cc:729] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbffa38f702a4e08b829ed069a47c103, State: Initialized, Role: FOLLOWER
I20260924 11:37:22.900053 22067 consensus_queue.cc:260] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbffa38f702a4e08b829ed069a47c103" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 40045 } } peers { permanent_uuid: "6cb7f6130bda4786bc162352b37ee6ca" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 43247 } } peers { permanent_uuid: "1d695defd9734c80a12887ed84ede67c" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 44015 } }
I20260924 11:37:22.901698 22067 ts_tablet_manager.cc:1428] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103: Time spent starting tablet: real 0.015s user 0.011s sys 0.003s
I20260924 11:37:22.908576 22011 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.1:40045
I20260924 11:37:22.908620 22175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.1:40045 every 8 connection(s)
I20260924 11:37:22.909801 22011 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
I20260924 11:37:22.912046 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 22011
I20260924 11:37:22.912675 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.2:43247
--local_ip_for_outbound_sockets=127.6.212.2
--tserver_master_addrs=127.6.212.62:37655
--webserver_port=35125
--webserver_interface=127.6.212.2
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab
--principal=kudu/127.6.212.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:36593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:37:22.914503 22178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:22.912496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.1:40625)
0924 11:37:22.913372 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:22.913639 (+ 267us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":378,"threads_started":1}
W20260924 11:37:22.915112 22176 heartbeater.cc:644] Failed to heartbeat to 127.6.212.62:37655 (0 consecutive failures): Network error: Failed to ping master at 127.6.212.62:37655: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
I20260924 11:37:22.916025 22178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:22.915717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.1:58097)
0924 11:37:22.915856 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:22.915928 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 11:37:22.916874 22178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:22.916542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.1:52657)
0924 11:37:22.916688 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:22.916763 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 11:37:22.917080 22176 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 11:37:23.252534 22179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:37:23.252866 22179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:37:23.252970 22179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:37:23.260521 22179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:37:23.260641 22179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:37:23.260756 22179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.2
I20260924 11:37:23.270938 22179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:36593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab
--principal=kudu/127.6.212.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.2:43247
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.212.2
--webserver_port=35125
--tserver_master_addrs=127.6.212.62:37655
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.2
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:37:23.272320 22179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:37:23.273674 22179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:37:23.287938 22184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:37:23.288010 22185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:37:23.287976 22187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:37:23.288758 22179 server_base.cc:1034] running on GCE node
I20260924 11:37:23.289568 22179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:37:23.290566 22179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:37:23.291800 22179 hybrid_clock.cc:648] HybridClock initialized: now 1790249843291748 us; error 56 us; skew 500 ppm
I20260924 11:37:23.292155 22179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:37:23 dist-test-slave-cq7c krb5kdc[20860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249843, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:37:23.295674 22179 init.cc:377] Logged in from keytab as kudu/127.6.212.2@KRBTEST.COM (short username kudu)
I20260924 11:37:23.297348 22179 webserver.cc:460] Webserver started at http://127.6.212.2:35125/ using document root <none> and password file <none>
I20260924 11:37:23.298133 22179 fs_manager.cc:359] Metadata directory not provided
I20260924 11:37:23.298281 22179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:37:23.304067 22179 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 11:37:23.307272 22194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:23.308820 22179 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:37:23.309022 22179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-1/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-1/wal
uuid: "6cb7f6130bda4786bc162352b37ee6ca"
format_stamp: "Formatted at 2026-09-24 11:37:17 on dist-test-slave-cq7c"
server_key: "e1642fcd032ee8456f94393532c660ee"
server_key_iv: "6bc1321ed194db0a5bc1e8395b4f0829"
server_key_version: "encryptionkey@0"
I20260924 11:37:23.310091 22179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:37:23.325779 22179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:37:23.330250 22179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:37:23.330541 22179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:37:23.331662 22179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:37:23.333076 22179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:37:23.333362 22179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:23.333515 22179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:37:23.333611 22179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:23.368848 22179 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.2:43247
I20260924 11:37:23.368942 22306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.2:43247 every 8 connection(s)
I20260924 11:37:23.370497 22179 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
I20260924 11:37:23.375764 22309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:23.373769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.2:58129)
0924 11:37:23.374492 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:23.374697 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":275,"threads_started":1}
W20260924 11:37:23.376451 22307 heartbeater.cc:644] Failed to heartbeat to 127.6.212.62:37655 (0 consecutive failures): Network error: Failed to ping master at 127.6.212.62:37655: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
I20260924 11:37:23.377311 22309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:23.377013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.2:60533)
0924 11:37:23.377146 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:23.377213 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 11:37:23.378026 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 22179
I20260924 11:37:23.378338 22309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:23.377942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.2:44563)
0924 11:37:23.378113 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:23.378208 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260924 11:37:23.378572 22307 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 11:37:23.378796 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.3:44015
--local_ip_for_outbound_sockets=127.6.212.3
--tserver_master_addrs=127.6.212.62:37655
--webserver_port=43683
--webserver_interface=127.6.212.3
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab
--principal=kudu/127.6.212.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:36593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:37:23.734946 22310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:37:23.735257 22310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:37:23.735481 22310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:37:23.743783 22310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:37:23.743907 22310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:37:23.744112 22310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.3
I20260924 11:37:23.754709 22310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:36593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab
--principal=kudu/127.6.212.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.3:44015
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.212.3
--webserver_port=43683
--tserver_master_addrs=127.6.212.62:37655
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.3
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:37:23.756105 22310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:37:23.757418 22310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:37:23.771329 22315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:37:23.771293 22316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:37:23.771351 22318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:37:23.772688 22310 server_base.cc:1034] running on GCE node
I20260924 11:37:23.773360 22310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:37:23.774405 22310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:37:23.775652 22310 hybrid_clock.cc:648] HybridClock initialized: now 1790249843775556 us; error 96 us; skew 500 ppm
I20260924 11:37:23.776052 22310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:37:23 dist-test-slave-cq7c krb5kdc[20860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249843, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:37:23.779458 22310 init.cc:377] Logged in from keytab as kudu/127.6.212.3@KRBTEST.COM (short username kudu)
I20260924 11:37:23.781169 22310 webserver.cc:460] Webserver started at http://127.6.212.3:43683/ using document root <none> and password file <none>
I20260924 11:37:23.782020 22310 fs_manager.cc:359] Metadata directory not provided
I20260924 11:37:23.782184 22310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:37:23.788059 22310 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 11:37:23.790879 22325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:23.792806 22310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 11:37:23.792984 22310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-2/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-2/wal
uuid: "1d695defd9734c80a12887ed84ede67c"
format_stamp: "Formatted at 2026-09-24 11:37:17 on dist-test-slave-cq7c"
server_key: "70770c8ecb3d8f8f6046c7648a791921"
server_key_iv: "dac51743039c1ff5ea727efd5dcac7ee"
server_key_version: "encryptionkey@0"
I20260924 11:37:23.793993 22310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:37:23.817811 22310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:37:23.822590 22310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:37:23.822855 22310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:37:23.824093 22310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:37:23.825449 22310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:37:23.825538 22310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:23.825691 22310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:37:23.825760 22310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:23.860658 22310 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.3:44015
I20260924 11:37:23.860872 22437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.3:44015 every 8 connection(s)
I20260924 11:37:23.862272 22310 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
I20260924 11:37:23.863353 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 22310
I20260924 11:37:23.867697 22441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:23.865764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.3:45681)
0924 11:37:23.866507 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:23.866732 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":284,"threads_started":1}
W20260924 11:37:23.868377 22438 heartbeater.cc:644] Failed to heartbeat to 127.6.212.62:37655 (0 consecutive failures): Network error: Failed to ping master at 127.6.212.62:37655: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
I20260924 11:37:23.869200 22441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:23.868895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.3:50883)
0924 11:37:23.869030 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:23.869098 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 11:37:23.870077 22441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:23.869752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.3:42327)
0924 11:37:23.869888 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:23.869970 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 11:37:23.870390 22438 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 11:37:23.918538 22442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:23.917743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.1:47775)
0924 11:37:23.918221 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:23.918316 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":252,"threads_started":1}
I20260924 11:37:24.380055 22443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:24.379193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.2:59601)
0924 11:37:24.379717 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:24.379832 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":291,"threads_started":1}
I20260924 11:37:24.727834 22444 raft_consensus.cc:482] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:37:24.728237 22444 raft_consensus.cc:504] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbffa38f702a4e08b829ed069a47c103" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 40045 } } peers { permanent_uuid: "6cb7f6130bda4786bc162352b37ee6ca" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 43247 } } peers { permanent_uuid: "1d695defd9734c80a12887ed84ede67c" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 44015 } }
I20260924 11:37:24.730031 22444 leader_election.cc:288] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6cb7f6130bda4786bc162352b37ee6ca (127.6.212.2:43247), 1d695defd9734c80a12887ed84ede67c (127.6.212.3:44015)
Sep 24 11:37:24 dist-test-slave-cq7c krb5kdc[20860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249842, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for kudu/127.6.212.2@KRBTEST.COM
Sep 24 11:37:24 dist-test-slave-cq7c krb5kdc[20860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249842, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for kudu/127.6.212.3@KRBTEST.COM
I20260924 11:37:24.771593 22445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:24.730352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.2:43247 (local address 127.6.212.1:45501)
0924 11:37:24.742520 (+ 12168us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:24.742538 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:37:24.743106 (+ 568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:37:24.746753 (+ 3647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:37:24.746765 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:37:24.748112 (+ 1347us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:37:24.749404 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:24.749435 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:24.751857 (+ 2422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:24.751862 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:24.754410 (+ 2548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:24.754427 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:24.756047 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:24.756054 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:24.756349 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:24.760899 (+ 4550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:37:24.760926 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:37:24.763752 (+ 2826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:37:24.767800 (+ 4048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:24.767812 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:24.767820 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:24.768378 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:24.768832 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:24.768837 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:24.768841 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:24.769045 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:24.770208 (+ 1163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:37:24.770216 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:37:24.770399 (+ 183us) client_negotiation.cc:769] Sending connection context
0924 11:37:24.770534 (+ 135us) client_negotiation.cc:240] Negotiation successful
0924 11:37:24.771056 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12024,"mutex_wait_us":505,"spinlock_wait_cycles":4608,"thread_start_us":11819,"threads_started":1}
I20260924 11:37:24.771934 22447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:24.730607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:45501 (local address 127.6.212.2:43247)
0924 11:37:24.741892 (+ 11285us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:24.741911 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:24.742648 (+ 737us) server_negotiation.cc:407] Connection header received
0924 11:37:24.745666 (+ 3018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:24.745675 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:24.746166 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:24.746546 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:24.749650 (+ 3104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:24.751619 (+ 1969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:24.754593 (+ 2974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:24.755753 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:24.756034 (+ 281us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:24.764152 (+ 8118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:24.764168 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:24.764200 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:24.764241 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:24.767630 (+ 3389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:24.768542 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:24.768549 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:24.768555 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:24.768706 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:24.769167 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:24.769172 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:24.769176 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:24.769725 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:24.769931 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:24.771125 (+ 1194us) server_negotiation.cc:299] Negotiation successful
0924 11:37:24.771572 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11096,"thread_start_us":10940,"threads_started":1}
I20260924 11:37:24.771991 22448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:24.730690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:41199 (local address 127.6.212.3:44015)
0924 11:37:24.741076 (+ 10386us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:24.741104 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:24.742629 (+ 1525us) server_negotiation.cc:407] Connection header received
0924 11:37:24.745626 (+ 2997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:24.745639 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:24.746166 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:24.746546 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:24.749649 (+ 3103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:24.751628 (+ 1979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:24.754601 (+ 2973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:24.755785 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:24.755973 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:24.764310 (+ 8337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:24.764319 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:24.764334 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:24.764373 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:24.767620 (+ 3247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:24.768542 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:24.768549 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:24.768555 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:24.768710 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:24.769206 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:24.769210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:24.769212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:24.769718 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:24.769914 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:24.771143 (+ 1229us) server_negotiation.cc:299] Negotiation successful
0924 11:37:24.771580 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10094,"thread_start_us":10460,"threads_started":1}
I20260924 11:37:24.771680 22446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:24.730352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.3:44015 (local address 127.6.212.1:41199)
0924 11:37:24.742357 (+ 12005us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:24.742427 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0924 11:37:24.743106 (+ 679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:37:24.746753 (+ 3647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:37:24.746765 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:37:24.748112 (+ 1347us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:37:24.749404 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:24.749435 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:24.751863 (+ 2428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:24.751867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:24.754410 (+ 2543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:24.754427 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:24.755999 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:24.756009 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:24.756355 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:24.760907 (+ 4552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:37:24.760935 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:37:24.764024 (+ 3089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:37:24.767791 (+ 3767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:24.767802 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:24.767818 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:24.768384 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:24.768847 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:24.768852 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:24.768856 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:24.769056 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:24.769870 (+ 814us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:37:24.769878 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:37:24.770197 (+ 319us) client_negotiation.cc:769] Sending connection context
0924 11:37:24.770503 (+ 306us) client_negotiation.cc:240] Negotiation successful
0924 11:37:24.771248 (+ 745us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11806,"mutex_wait_us":561,"spinlock_wait_cycles":156160,"thread_start_us":11550,"threads_started":1}
W20260924 11:37:24.775138 22063 leader_election.cc:341] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6cb7f6130bda4786bc162352b37ee6ca (127.6.212.2:43247): Not found: Tablet not found: 8649defef81d4cf5bfbd406fe54378e9
W20260924 11:37:24.775405 22062 leader_election.cc:341] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d695defd9734c80a12887ed84ede67c (127.6.212.3:44015): Not found: Tablet not found: 8649defef81d4cf5bfbd406fe54378e9
I20260924 11:37:24.775579 22062 leader_election.cc:302] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbffa38f702a4e08b829ed069a47c103; no voters: 1d695defd9734c80a12887ed84ede67c, 6cb7f6130bda4786bc162352b37ee6ca
I20260924 11:37:24.775998 22444 raft_consensus.cc:2726] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 11:37:24.872079 22449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:24.871011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.3:58991)
0924 11:37:24.871620 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:24.871822 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":315,"threads_started":1}
I20260924 11:37:24.920035 22446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:24.919584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.1:54459)
0924 11:37:24.919741 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:24.919844 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 11:37:25.381649 22450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:25.380946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.2:39893)
0924 11:37:25.381411 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:25.381491 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":261,"threads_started":1}
I20260924 11:37:25.873744 22451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:25.872942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.3:60817)
0924 11:37:25.873444 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:25.873521 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":291,"threads_started":1}
I20260924 11:37:25.921738 22452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:25.920987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.1:54827)
0924 11:37:25.921473 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:25.921553 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":267,"threads_started":1}
I20260924 11:37:26.383802 22453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:26.382774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.2:56599)
0924 11:37:26.383433 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:26.383541 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":304,"threads_started":1}
I20260924 11:37:26.841478 22454 raft_consensus.cc:482] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:37:26.841666 22454 raft_consensus.cc:504] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbffa38f702a4e08b829ed069a47c103" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 40045 } } peers { permanent_uuid: "6cb7f6130bda4786bc162352b37ee6ca" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 43247 } } peers { permanent_uuid: "1d695defd9734c80a12887ed84ede67c" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 44015 } }
I20260924 11:37:26.842271 22454 leader_election.cc:288] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6cb7f6130bda4786bc162352b37ee6ca (127.6.212.2:43247), 1d695defd9734c80a12887ed84ede67c (127.6.212.3:44015)
W20260924 11:37:26.843482 22063 leader_election.cc:341] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6cb7f6130bda4786bc162352b37ee6ca (127.6.212.2:43247): Not found: Tablet not found: 8649defef81d4cf5bfbd406fe54378e9
W20260924 11:37:26.843704 22062 leader_election.cc:341] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d695defd9734c80a12887ed84ede67c (127.6.212.3:44015): Not found: Tablet not found: 8649defef81d4cf5bfbd406fe54378e9
I20260924 11:37:26.843801 22062 leader_election.cc:302] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbffa38f702a4e08b829ed069a47c103; no voters: 1d695defd9734c80a12887ed84ede67c, 6cb7f6130bda4786bc162352b37ee6ca
I20260924 11:37:26.844018 22454 raft_consensus.cc:2726] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 11:37:26.875622 22455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:26.874766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.3:52493)
0924 11:37:26.875265 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:26.875352 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":243,"threads_started":1}
I20260924 11:37:26.923712 22456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:26.922805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.1:59309)
0924 11:37:26.923318 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:26.923501 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":236,"threads_started":1}
I20260924 11:37:27.385438 22457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:27.384721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.2:54407)
0924 11:37:27.385150 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:27.385238 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":204,"threads_started":1}
I20260924 11:37:27.877324 22458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:27.876582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.3:36697)
0924 11:37:27.877026 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:27.877107 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":231,"threads_started":1}
I20260924 11:37:27.925629 22459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:27.924818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.1:45641)
0924 11:37:27.925350 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:27.925442 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":261,"threads_started":1}
I20260924 11:37:28.387115 22460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:28.386358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.2:36721)
0924 11:37:28.386869 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:28.386953 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":242,"threads_started":1}
I20260924 11:37:28.879474 22461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:28.878520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.3:46969)
0924 11:37:28.879122 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:28.879212 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":370,"threads_started":1}
I20260924 11:37:28.927417 22462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:28.926624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.1:60581)
0924 11:37:28.927089 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:28.927190 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":265,"threads_started":1}
I20260924 11:37:29.389128 22463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:29.388259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.2:44065)
0924 11:37:29.388805 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:29.388913 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":306,"threads_started":1}
I20260924 11:37:29.881186 22464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:29.880474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.3:43585)
0924 11:37:29.880923 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:29.881006 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":241,"threads_started":1}
I20260924 11:37:29.908393 22465 raft_consensus.cc:482] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:37:29.908659 22465 raft_consensus.cc:504] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbffa38f702a4e08b829ed069a47c103" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 40045 } } peers { permanent_uuid: "6cb7f6130bda4786bc162352b37ee6ca" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 43247 } } peers { permanent_uuid: "1d695defd9734c80a12887ed84ede67c" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 44015 } }
I20260924 11:37:29.909425 22465 leader_election.cc:288] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6cb7f6130bda4786bc162352b37ee6ca (127.6.212.2:43247), 1d695defd9734c80a12887ed84ede67c (127.6.212.3:44015)
W20260924 11:37:29.910360 22062 leader_election.cc:341] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d695defd9734c80a12887ed84ede67c (127.6.212.3:44015): Not found: Tablet not found: 8649defef81d4cf5bfbd406fe54378e9
W20260924 11:37:29.910604 22063 leader_election.cc:341] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6cb7f6130bda4786bc162352b37ee6ca (127.6.212.2:43247): Not found: Tablet not found: 8649defef81d4cf5bfbd406fe54378e9
I20260924 11:37:29.910777 22063 leader_election.cc:302] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbffa38f702a4e08b829ed069a47c103; no voters: 1d695defd9734c80a12887ed84ede67c, 6cb7f6130bda4786bc162352b37ee6ca
I20260924 11:37:29.911057 22465 raft_consensus.cc:2726] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 11:37:29.929409 22466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:29.928514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.1:35511)
0924 11:37:29.929087 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:29.929185 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":335,"threads_started":1}
I20260924 11:37:30.391129 22467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:30.390197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.2:32783)
0924 11:37:30.390822 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:30.390926 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":264,"threads_started":1}
I20260924 11:37:30.883041 22468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:30.882201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.3:43021)
0924 11:37:30.882748 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:30.882837 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":231,"threads_started":1}
I20260924 11:37:30.931056 22469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:30.930353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.1:58993)
0924 11:37:30.930804 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:30.930888 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":225,"threads_started":1}
I20260924 11:37:31.393085 22470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:31.392323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.2:34469)
0924 11:37:31.392799 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:31.392890 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":240,"threads_started":1}
I20260924 11:37:31.885170 22471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:31.884213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.3:48199)
0924 11:37:31.884871 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:31.884963 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":361,"threads_started":1}
I20260924 11:37:31.932947 22472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:31.932067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.1:35311)
0924 11:37:31.932617 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:31.932722 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":277,"threads_started":1}
I20260924 11:37:32.395009 22473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:32.394231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.2:60685)
0924 11:37:32.394738 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:32.394845 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":244,"threads_started":1}
I20260924 11:37:32.809240 22474 raft_consensus.cc:482] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:37:32.809420 22474 raft_consensus.cc:504] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbffa38f702a4e08b829ed069a47c103" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 40045 } } peers { permanent_uuid: "6cb7f6130bda4786bc162352b37ee6ca" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 43247 } } peers { permanent_uuid: "1d695defd9734c80a12887ed84ede67c" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 44015 } }
I20260924 11:37:32.810087 22474 leader_election.cc:288] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6cb7f6130bda4786bc162352b37ee6ca (127.6.212.2:43247), 1d695defd9734c80a12887ed84ede67c (127.6.212.3:44015)
W20260924 11:37:32.811147 22063 leader_election.cc:341] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6cb7f6130bda4786bc162352b37ee6ca (127.6.212.2:43247): Not found: Tablet not found: 8649defef81d4cf5bfbd406fe54378e9
W20260924 11:37:32.811318 22062 leader_election.cc:341] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d695defd9734c80a12887ed84ede67c (127.6.212.3:44015): Not found: Tablet not found: 8649defef81d4cf5bfbd406fe54378e9
I20260924 11:37:32.811496 22062 leader_election.cc:302] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbffa38f702a4e08b829ed069a47c103; no voters: 1d695defd9734c80a12887ed84ede67c, 6cb7f6130bda4786bc162352b37ee6ca
I20260924 11:37:32.811810 22474 raft_consensus.cc:2726] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 11:37:32.887037 22475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:32.886187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.3:36733)
0924 11:37:32.886751 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:32.886857 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":237,"threads_started":1}
I20260924 11:37:32.934672 22476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:32.933918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.1:55733)
0924 11:37:32.934400 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:32.934480 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":245,"threads_started":1}
I20260924 11:37:33.396814 22477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:33.396066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.2:36237)
0924 11:37:33.396532 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:33.396618 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":258,"threads_started":1}
I20260924 11:37:33.889132 22478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:33.888231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.3:42807)
0924 11:37:33.888815 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:33.888905 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":265,"threads_started":1}
I20260924 11:37:33.936483 22479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:33.935736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.1:53077)
0924 11:37:33.936215 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:33.936310 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":254,"threads_started":1}
I20260924 11:37:34.398612 22480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:34.397840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.2:57827)
0924 11:37:34.398363 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:34.398440 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":269,"threads_started":1}
I20260924 11:37:34.890947 22481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:34.890200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.3:33499)
0924 11:37:34.890683 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:34.890783 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":233,"threads_started":1}
I20260924 11:37:34.938252 22482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:34.937468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.1:58221)
0924 11:37:34.937994 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:34.938083 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":311,"threads_started":1}
I20260924 11:37:35.400333 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:35.399606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.2:35291)
0924 11:37:35.400044 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:35.400127 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":222,"threads_started":1}
I20260924 11:37:35.504077 22484 raft_consensus.cc:482] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:37:35.504310 22484 raft_consensus.cc:504] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbffa38f702a4e08b829ed069a47c103" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 40045 } } peers { permanent_uuid: "6cb7f6130bda4786bc162352b37ee6ca" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 43247 } } peers { permanent_uuid: "1d695defd9734c80a12887ed84ede67c" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 44015 } }
I20260924 11:37:35.505116 22484 leader_election.cc:288] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6cb7f6130bda4786bc162352b37ee6ca (127.6.212.2:43247), 1d695defd9734c80a12887ed84ede67c (127.6.212.3:44015)
W20260924 11:37:35.506189 22063 leader_election.cc:341] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6cb7f6130bda4786bc162352b37ee6ca (127.6.212.2:43247): Not found: Tablet not found: 8649defef81d4cf5bfbd406fe54378e9
W20260924 11:37:35.506351 22062 leader_election.cc:341] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d695defd9734c80a12887ed84ede67c (127.6.212.3:44015): Not found: Tablet not found: 8649defef81d4cf5bfbd406fe54378e9
I20260924 11:37:35.506517 22062 leader_election.cc:302] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbffa38f702a4e08b829ed069a47c103; no voters: 1d695defd9734c80a12887ed84ede67c, 6cb7f6130bda4786bc162352b37ee6ca
I20260924 11:37:35.506834 22484 raft_consensus.cc:2726] T 8649defef81d4cf5bfbd406fe54378e9 P bbffa38f702a4e08b829ed069a47c103 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 11:37:35.892923 22485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:35.892039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.3:37293)
0924 11:37:35.892621 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:35.892728 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":324,"threads_started":1}
I20260924 11:37:35.940196 22486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:35.939244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.1:37151)
0924 11:37:35.939828 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:35.939948 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":344,"threads_started":1}
I20260924 11:37:36.402258 22487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:36.401312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.2:60411)
0924 11:37:36.401954 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:36.402091 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":369,"threads_started":1}
I20260924 11:37:36.894810 22488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:36.894000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.3:57005)
0924 11:37:36.894515 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:36.894617 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":266,"threads_started":1}
I20260924 11:37:36.942190 22489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:36.941320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.1:37709)
0924 11:37:36.941863 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:36.941971 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":324,"threads_started":1}
I20260924 11:37:37.404232 22490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:37.403291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.2:46375)
0924 11:37:37.403959 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:37.404043 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":448,"threads_started":1}
I20260924 11:37:37.896956 22491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:37.896042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.3:59181)
0924 11:37:37.896521 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:37.896670 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":233,"threads_started":1}
I20260924 11:37:37.944499 22492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:37.943633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.1:56679)
0924 11:37:37.944218 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:37.944310 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":273,"threads_started":1}
I20260924 11:37:38.406239 22493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:38.405472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.2:46735)
0924 11:37:38.405970 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:38.406063 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":241,"threads_started":1}
I20260924 11:37:38.867513 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 22011
I20260924 11:37:38.890779 22494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:38.868952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38730 (local address 127.6.212.1:40045)
0924 11:37:38.869599 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:38.869608 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:38.869638 (+ 30us) server_negotiation.cc:407] Connection header received
0924 11:37:38.869713 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:38.869720 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:38.869962 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:38.870160 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:38.872704 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:38.873760 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:38.875193 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:38.875962 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:38.876128 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:38.884242 (+ 8114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:38.884259 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:38.884275 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:38.884319 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:38.887518 (+ 3199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:38.888394 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:38.888408 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:38.888411 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:38.888571 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:38.888987 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:38.888993 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:38.888997 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:38.889433 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:38.889793 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:38.890116 (+ 323us) server_negotiation.cc:299] Negotiation successful
0924 11:37:38.890415 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":318,"threads_started":1}
I20260924 11:37:38.892233 22171 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:37:38.898859 22496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:38.898079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.3:46861)
0924 11:37:38.898583 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:38.898698 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":280,"threads_started":1}
I20260924 11:37:38.988186 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 22011
I20260924 11:37:38.988271 22497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:38.983605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:37655 (local address 127.6.212.1:45207)
0924 11:37:38.987798 (+ 4193us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:38.987911 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:37655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4071,"thread_start_us":3927,"threads_started":1}
I20260924 11:37:39.004328 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 22179
I20260924 11:37:39.019122 22498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:39.005127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44512 (local address 127.6.212.2:43247)
0924 11:37:39.005698 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:39.005702 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:39.005719 (+ 17us) server_negotiation.cc:407] Connection header received
0924 11:37:39.005811 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:39.005815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:39.005874 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:39.005964 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:39.007314 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:39.008171 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:39.009403 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:39.009915 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:39.010028 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:39.012992 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:39.012997 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:39.013001 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:39.013039 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:39.016373 (+ 3334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:39.017222 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:39.017226 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:39.017229 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:39.017387 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:39.017839 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:39.017842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:39.017844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:39.018125 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:39.018337 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:39.018626 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 11:37:39.018808 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":347,"threads_started":1}
I20260924 11:37:39.019968 22302 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:37:39.107726 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 22179
I20260924 11:37:39.126415 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 22310
I20260924 11:37:39.140714 22500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:39.127321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59276 (local address 127.6.212.3:44015)
0924 11:37:39.127894 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:39.127899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:39.127917 (+ 18us) server_negotiation.cc:407] Connection header received
0924 11:37:39.127977 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:39.127981 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:39.128051 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:39.128150 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:39.129552 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:39.130355 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:39.131685 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:39.132108 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:39.132205 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:39.134946 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:39.134951 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:39.134955 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:39.134988 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:39.137811 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:39.138444 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:39.138449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:39.138451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:39.138594 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:39.138990 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:39.138996 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:39.138999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:39.139285 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:39.139628 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:39.140143 (+ 515us) server_negotiation.cc:299] Negotiation successful
0924 11:37:39.140359 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":339,"threads_started":1}
I20260924 11:37:39.141507 22433 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:37:39.227782 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 22310
2026-09-24T11:37:39Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24948 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:37:39 dist-test-slave-cq7c krb5kdc[22504](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:37:39 dist-test-slave-cq7c krb5kdc[22504](info): set up 2 sockets
Sep 24 11:37:39 dist-test-slave-cq7c krb5kdc[22504](info): commencing operation
krb5kdc: starting...
W20260924 11:37:40.400816 6992 mini_kdc.cc:121] Time spent starting KDC: real 1.132s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Sep 24 11:37:40 dist-test-slave-cq7c krb5kdc[22504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:37:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:37:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.212.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
I20260924 11:37:40.700604 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:36825
--webserver_interface=127.6.212.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:36245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:36825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:37:41.062883 22517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:37:41.063266 22517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:37:41.063472 22517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:37:41.071177 22517 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 11:37:41.071336 22517 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 11:37:41.071434 22517 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 11:37:41.071547 22517 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:37:41.071640 22517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:37:41.071768 22517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:37:41.071859 22517 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:37:41.071939 22517 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:37:41.083338 22517 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:36245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:36825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:36825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.212.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:37:41.084880 22517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:37:41.086411 22517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:37:41.100128 22522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:37:41.100085 22525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:37:41.100044 22523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:37:41.100889 22517 server_base.cc:1034] running on GCE node
I20260924 11:37:41.101815 22517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:37:41.103314 22517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:37:41.104602 22517 hybrid_clock.cc:648] HybridClock initialized: now 1790249861104522 us; error 55 us; skew 500 ppm
I20260924 11:37:41.104993 22517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:37:41 dist-test-slave-cq7c krb5kdc[22504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249861, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:37:41.108256 22517 init.cc:377] Logged in from keytab as kudu/127.6.212.62@KRBTEST.COM (short username kudu)
I20260924 11:37:41.109881 22517 webserver.cc:460] Webserver started at http://127.6.212.62:36195/ using document root <none> and password file <none>
I20260924 11:37:41.110641 22517 fs_manager.cc:359] Metadata directory not provided
I20260924 11:37:41.110780 22517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:37:41.111111 22517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:37:41.114365 22517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "ac21188da6be46928a99557e485926bf"
format_stamp: "Formatted at 2026-09-24 11:37:41 on dist-test-slave-cq7c"
server_key: "c9a674c6ddf3b9c9bfe32ff0b221b7c1"
server_key_iv: "5aeadd200445eafd60a4ecce6a1dfaa2"
server_key_version: "encryptionkey@0"
I20260924 11:37:41.115208 22517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "ac21188da6be46928a99557e485926bf"
format_stamp: "Formatted at 2026-09-24 11:37:41 on dist-test-slave-cq7c"
server_key: "c9a674c6ddf3b9c9bfe32ff0b221b7c1"
server_key_iv: "5aeadd200445eafd60a4ecce6a1dfaa2"
server_key_version: "encryptionkey@0"
I20260924 11:37:41.120684 22517 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 11:37:41.124584 22532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:41.126318 22517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 11:37:41.126559 22517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "ac21188da6be46928a99557e485926bf"
format_stamp: "Formatted at 2026-09-24 11:37:41 on dist-test-slave-cq7c"
server_key: "c9a674c6ddf3b9c9bfe32ff0b221b7c1"
server_key_iv: "5aeadd200445eafd60a4ecce6a1dfaa2"
server_key_version: "encryptionkey@0"
I20260924 11:37:41.126842 22517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:37:41.141022 22517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:37:41.146165 22517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:37:41.146497 22517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:37:41.166991 22517 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.62:36825
I20260924 11:37:41.167008 22583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.62:36825 every 8 connection(s)
I20260924 11:37:41.168689 22517 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:37:41.172652 22584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:37:41.178121 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 22517
I20260924 11:37:41.178390 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:37:41.179137 6992 external_mini_cluster.cc:1428] Setting key e38c5eecf7d993e395c905da980b9deb
I20260924 11:37:41.184952 22584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf: Bootstrap starting.
I20260924 11:37:41.189138 22584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf: Neither blocks nor log segments found. Creating new log.
I20260924 11:37:41.190588 22584 log.cc:824] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf: Log is configured to *not* fsync() on all Append() calls
I20260924 11:37:41.194519 22584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf: No bootstrap required, opened a new log
Sep 24 11:37:41 dist-test-slave-cq7c krb5kdc[22504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:37:41.205444 22584 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac21188da6be46928a99557e485926bf" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36825 } }
I20260924 11:37:41.205962 22584 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:37:41.206133 22584 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac21188da6be46928a99557e485926bf, State: Initialized, Role: FOLLOWER
I20260924 11:37:41.206878 22584 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac21188da6be46928a99557e485926bf" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36825 } }
I20260924 11:37:41.207214 22584 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:37:41.207392 22584 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:37:41.207599 22584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:37:41.208069 22587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:41.181424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41020 (local address 127.6.212.62:36825)
0924 11:37:41.182423 (+ 999us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:41.182438 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:41.182486 (+ 48us) server_negotiation.cc:407] Connection header received
0924 11:37:41.184386 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:41.184411 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:41.184869 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:41.185311 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:41.187252 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:41.188972 (+ 1720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:41.191099 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:41.192364 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:41.192508 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:41.197790 (+ 5282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:41.197802 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:41.197833 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:41.197890 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:41.202352 (+ 4462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:41.203383 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:41.203393 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:41.203404 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:41.203693 (+ 289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:41.204127 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:41.204131 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:41.204134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:41.204945 (+ 811us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:41.205252 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:41.205885 (+ 633us) server_negotiation.cc:299] Negotiation successful
0924 11:37:41.206409 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":718,"thread_start_us":440,"threads_started":1}
I20260924 11:37:41.210708 22584 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac21188da6be46928a99557e485926bf" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36825 } }
I20260924 11:37:41.211338 22584 leader_election.cc:302] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac21188da6be46928a99557e485926bf; no voters:
I20260924 11:37:41.212132 22584 leader_election.cc:288] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:37:41.212312 22589 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:37:41.213086 22589 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [term 1 LEADER]: Becoming Leader. State: Replica: ac21188da6be46928a99557e485926bf, State: Running, Role: LEADER
I20260924 11:37:41.213752 22589 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac21188da6be46928a99557e485926bf" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36825 } }
I20260924 11:37:41.214313 22584 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:37:41.217180 22591 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac21188da6be46928a99557e485926bf. Latest consensus state: current_term: 1 leader_uuid: "ac21188da6be46928a99557e485926bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac21188da6be46928a99557e485926bf" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36825 } } }
I20260924 11:37:41.217693 22591 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [sys.catalog]: This master's current role is: LEADER
I20260924 11:37:41.218340 22590 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac21188da6be46928a99557e485926bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac21188da6be46928a99557e485926bf" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36825 } } }
I20260924 11:37:41.218717 22590 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [sys.catalog]: This master's current role is: LEADER
I20260924 11:37:41.218925 22597 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:37:41.228336 22597 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:37:41.239880 22597 catalog_manager.cc:1347] Generated new cluster ID: 0b046aa47f3c42a38c8a9b1a5dcd2232
I20260924 11:37:41.240036 22597 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:37:41.263587 22597 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:37:41.265707 22597 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:37:41.277514 22597 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf: Generated new TSK 0
I20260924 11:37:41.278697 22597 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.212.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab.
Entry for principal kudu/127.6.212.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab.
I20260924 11:37:41.356390 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.1:0
--local_ip_for_outbound_sockets=127.6.212.1
--webserver_interface=127.6.212.1
--webserver_port=0
--tserver_master_addrs=127.6.212.62:36825
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab
--principal=kudu/127.6.212.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:36245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:37:41.734643 22610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:37:41.734993 22610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:37:41.735180 22610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:37:41.742897 22610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:37:41.743076 22610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:37:41.743257 22610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.1
I20260924 11:37:41.753558 22610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:36245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab
--principal=kudu/127.6.212.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.212.1
--webserver_port=0
--tserver_master_addrs=127.6.212.62:36825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.1
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:37:41.754992 22610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:37:41.756366 22610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:37:41.770987 22615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:37:41.770869 22618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:37:41.770951 22616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:37:41.772004 22610 server_base.cc:1034] running on GCE node
I20260924 11:37:41.772702 22610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:37:41.773820 22610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:37:41.775061 22610 hybrid_clock.cc:648] HybridClock initialized: now 1790249861774972 us; error 86 us; skew 500 ppm
I20260924 11:37:41.775460 22610 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:37:41 dist-test-slave-cq7c krb5kdc[22504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249861, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:37:41.778982 22610 init.cc:377] Logged in from keytab as kudu/127.6.212.1@KRBTEST.COM (short username kudu)
I20260924 11:37:41.780772 22610 webserver.cc:460] Webserver started at http://127.6.212.1:41729/ using document root <none> and password file <none>
I20260924 11:37:41.781526 22610 fs_manager.cc:359] Metadata directory not provided
I20260924 11:37:41.781682 22610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:37:41.782004 22610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:37:41.785252 22610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-0/data/instance:
uuid: "62c2c7c8728542e5a9f8f9851794a98d"
format_stamp: "Formatted at 2026-09-24 11:37:41 on dist-test-slave-cq7c"
server_key: "27ba1496484442fc22dfffff75b3fcc8"
server_key_iv: "95acc6f22192c558b523389e9c3e5bab"
server_key_version: "encryptionkey@0"
I20260924 11:37:41.786187 22610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance:
uuid: "62c2c7c8728542e5a9f8f9851794a98d"
format_stamp: "Formatted at 2026-09-24 11:37:41 on dist-test-slave-cq7c"
server_key: "27ba1496484442fc22dfffff75b3fcc8"
server_key_iv: "95acc6f22192c558b523389e9c3e5bab"
server_key_version: "encryptionkey@0"
I20260924 11:37:41.791953 22610 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 11:37:41.795804 22625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:41.797222 22610 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 11:37:41.797415 22610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-0/wal
uuid: "62c2c7c8728542e5a9f8f9851794a98d"
format_stamp: "Formatted at 2026-09-24 11:37:41 on dist-test-slave-cq7c"
server_key: "27ba1496484442fc22dfffff75b3fcc8"
server_key_iv: "95acc6f22192c558b523389e9c3e5bab"
server_key_version: "encryptionkey@0"
I20260924 11:37:41.797650 22610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:37:41.817476 22610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:37:41.822559 22610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:37:41.822883 22610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:37:41.824009 22610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:37:41.825505 22610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:37:41.825578 22610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:41.825695 22610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:37:41.825760 22610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:41.860752 22610 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.1:46119
I20260924 11:37:41.860863 22737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.1:46119 every 8 connection(s)
I20260924 11:37:41.862177 22610 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
I20260924 11:37:41.866374 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 22610
I20260924 11:37:41.866628 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance
I20260924 11:37:41.867519 6992 external_mini_cluster.cc:1428] Setting key 0d903ebc626e68d608f5d5d55f99d6e2
Sep 24 11:37:41 dist-test-slave-cq7c krb5kdc[22504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249861, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:37:41.891228 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:41.865045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:42581 (local address 127.6.212.62:36825)
0924 11:37:41.865528 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:41.865533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:41.866916 (+ 1383us) server_negotiation.cc:407] Connection header received
0924 11:37:41.870730 (+ 3814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:41.870735 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:41.870807 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:41.870905 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:41.873412 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:41.874240 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:41.876980 (+ 2740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:41.877410 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:41.877591 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:41.882501 (+ 4910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:41.882513 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:41.882521 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:41.882580 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:41.886836 (+ 4256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:41.887993 (+ 1157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:41.888002 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:41.888009 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:41.888208 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:41.888804 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:41.888813 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:41.888820 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:41.889295 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:41.889479 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:41.890360 (+ 881us) server_negotiation.cc:299] Negotiation successful
0924 11:37:41.890664 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":261,"threads_started":1}
I20260924 11:37:41.891881 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:41.865719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.1:42581)
0924 11:37:41.866688 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:41.866752 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 11:37:41.870005 (+ 3253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:37:41.871273 (+ 1268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:37:41.871290 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:37:41.872220 (+ 930us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:37:41.873180 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:41.873211 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:41.874454 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:41.874462 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:41.876790 (+ 2328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:41.876809 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:41.877674 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:41.877684 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:41.878024 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:41.878942 (+ 918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:37:41.878985 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:37:41.882161 (+ 3176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:37:41.887091 (+ 4930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:41.887101 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:41.887119 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:41.887807 (+ 688us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:41.888389 (+ 582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:41.888394 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:41.888396 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:41.888633 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:41.889459 (+ 826us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:37:41.889466 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:37:41.889869 (+ 403us) client_negotiation.cc:769] Sending connection context
0924 11:37:41.890163 (+ 294us) client_negotiation.cc:240] Negotiation successful
0924 11:37:41.890620 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":375,"threads_started":1}
I20260924 11:37:41.893720 22738 heartbeater.cc:347] Connected to a master server at 127.6.212.62:36825
I20260924 11:37:41.894047 22738 heartbeater.cc:464] Registering TS with master...
I20260924 11:37:41.894802 22738 heartbeater.cc:511] Master 127.6.212.62:36825 requested a full tablet report, sending...
I20260924 11:37:41.897094 22549 ts_manager.cc:194] Registered new tserver with Master: 62c2c7c8728542e5a9f8f9851794a98d (127.6.212.1:46119)
WARNING: no policy specified for kudu/127.6.212.2@KRBTEST.COM; defaulting to no policy
I20260924 11:37:41.898705 22549 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.1@KRBTEST.COM'} at 127.6.212.1:42581
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab.
Entry for principal kudu/127.6.212.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab.
I20260924 11:37:41.935428 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.2:0
--local_ip_for_outbound_sockets=127.6.212.2
--webserver_interface=127.6.212.2
--webserver_port=0
--tserver_master_addrs=127.6.212.62:36825
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab
--principal=kudu/127.6.212.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:36245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:37:42.286332 22744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:37:42.286688 22744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:37:42.286788 22744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:37:42.294775 22744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:37:42.294976 22744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:37:42.295193 22744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.2
I20260924 11:37:42.305713 22744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:36245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab
--principal=kudu/127.6.212.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.212.2
--webserver_port=0
--tserver_master_addrs=127.6.212.62:36825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.2
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:37:42.307009 22744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:37:42.308461 22744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:37:42.322918 22749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:37:42.322944 22750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:37:42.322903 22752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:37:42.323828 22744 server_base.cc:1034] running on GCE node
I20260924 11:37:42.324488 22744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:37:42.325579 22744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:37:42.326848 22744 hybrid_clock.cc:648] HybridClock initialized: now 1790249862326798 us; error 35 us; skew 500 ppm
I20260924 11:37:42.327229 22744 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:37:42 dist-test-slave-cq7c krb5kdc[22504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249862, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:37:42.330741 22744 init.cc:377] Logged in from keytab as kudu/127.6.212.2@KRBTEST.COM (short username kudu)
I20260924 11:37:42.332777 22744 webserver.cc:460] Webserver started at http://127.6.212.2:41293/ using document root <none> and password file <none>
I20260924 11:37:42.333526 22744 fs_manager.cc:359] Metadata directory not provided
I20260924 11:37:42.333633 22744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:37:42.333910 22744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:37:42.337118 22744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-1/data/instance:
uuid: "63e7d703fab14977bd19cb32ff984f5e"
format_stamp: "Formatted at 2026-09-24 11:37:42 on dist-test-slave-cq7c"
server_key: "e97b9476f1007d167819ee0e85b6c5a6"
server_key_iv: "ffb18584d4aa4d09a8742220718447aa"
server_key_version: "encryptionkey@0"
I20260924 11:37:42.337864 22744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance:
uuid: "63e7d703fab14977bd19cb32ff984f5e"
format_stamp: "Formatted at 2026-09-24 11:37:42 on dist-test-slave-cq7c"
server_key: "e97b9476f1007d167819ee0e85b6c5a6"
server_key_iv: "ffb18584d4aa4d09a8742220718447aa"
server_key_version: "encryptionkey@0"
I20260924 11:37:42.343523 22744 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 11:37:42.347167 22759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:42.348834 22744 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:37:42.349061 22744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-1/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-1/wal
uuid: "63e7d703fab14977bd19cb32ff984f5e"
format_stamp: "Formatted at 2026-09-24 11:37:42 on dist-test-slave-cq7c"
server_key: "e97b9476f1007d167819ee0e85b6c5a6"
server_key_iv: "ffb18584d4aa4d09a8742220718447aa"
server_key_version: "encryptionkey@0"
I20260924 11:37:42.349261 22744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:37:42.376566 22744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:37:42.381141 22744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:37:42.381471 22744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:37:42.382520 22744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:37:42.384006 22744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:37:42.384109 22744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:42.384268 22744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:37:42.384341 22744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:42.421936 22744 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.2:36853
I20260924 11:37:42.422039 22871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.2:36853 every 8 connection(s)
I20260924 11:37:42.423440 22744 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
I20260924 11:37:42.432498 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 22744
I20260924 11:37:42.432693 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance
I20260924 11:37:42.433233 6992 external_mini_cluster.cc:1428] Setting key c351be5cdb2a573c5233c424af9cef8c
Sep 24 11:37:42 dist-test-slave-cq7c krb5kdc[22504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249862, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.2@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:37:42.447649 22874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:42.426310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:48379 (local address 127.6.212.62:36825)
0924 11:37:42.426781 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:42.426786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:42.427817 (+ 1031us) server_negotiation.cc:407] Connection header received
0924 11:37:42.429698 (+ 1881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:42.429704 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:42.429799 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:42.429919 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:42.431777 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:42.432873 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:42.434804 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:42.435592 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:42.435743 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:42.439573 (+ 3830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:42.439582 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:42.439590 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:42.439636 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:42.443664 (+ 4028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:42.444577 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:42.444586 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:42.444593 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:42.444821 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:42.445530 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:42.445534 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:42.445536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:42.445854 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:42.445985 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:42.447058 (+ 1073us) server_negotiation.cc:299] Negotiation successful
0924 11:37:42.447273 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":261,"threads_started":1}
I20260924 11:37:42.449491 22875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:42.426809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.2:48379)
0924 11:37:42.427658 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:42.427704 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 11:37:42.429410 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:37:42.430168 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:37:42.430176 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:37:42.430827 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:37:42.431598 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:42.431619 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:42.433069 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:42.433073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:42.434650 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:42.434669 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:42.435835 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:42.435840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:42.436052 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:42.436790 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:37:42.436813 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:37:42.439286 (+ 2473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:37:42.443853 (+ 4567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:42.443863 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:42.443878 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:42.444413 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:42.445011 (+ 598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:42.445020 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:42.445025 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:42.445378 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:42.446011 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:37:42.446024 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:37:42.446614 (+ 590us) client_negotiation.cc:769] Sending connection context
0924 11:37:42.446966 (+ 352us) client_negotiation.cc:240] Negotiation successful
0924 11:37:42.447722 (+ 756us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":302,"threads_started":1}
I20260924 11:37:42.451267 22872 heartbeater.cc:347] Connected to a master server at 127.6.212.62:36825
I20260924 11:37:42.451661 22872 heartbeater.cc:464] Registering TS with master...
I20260924 11:37:42.452519 22872 heartbeater.cc:511] Master 127.6.212.62:36825 requested a full tablet report, sending...
I20260924 11:37:42.454335 22549 ts_manager.cc:194] Registered new tserver with Master: 63e7d703fab14977bd19cb32ff984f5e (127.6.212.2:36853)
I20260924 11:37:42.455296 22549 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.2@KRBTEST.COM'} at 127.6.212.2:48379
WARNING: no policy specified for kudu/127.6.212.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab.
Entry for principal kudu/127.6.212.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab.
I20260924 11:37:42.499622 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.3:0
--local_ip_for_outbound_sockets=127.6.212.3
--webserver_interface=127.6.212.3
--webserver_port=0
--tserver_master_addrs=127.6.212.62:36825
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab
--principal=kudu/127.6.212.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:36245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:37:42.854841 22878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:37:42.855171 22878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:37:42.855340 22878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:37:42.862887 22878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:37:42.863075 22878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:37:42.863272 22878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.3
I20260924 11:37:42.873968 22878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:36245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab
--principal=kudu/127.6.212.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.212.3
--webserver_port=0
--tserver_master_addrs=127.6.212.62:36825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.3
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:37:42.875541 22878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:37:42.876921 22878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:37:42.890933 22883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:37:42.890961 22886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:37:42.891016 22884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:37:42.891863 22878 server_base.cc:1034] running on GCE node
I20260924 11:37:42.892575 22878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:37:42.893636 22878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:37:42.894868 22878 hybrid_clock.cc:648] HybridClock initialized: now 1790249862894815 us; error 49 us; skew 500 ppm
I20260924 11:37:42.895205 22878 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:37:42 dist-test-slave-cq7c krb5kdc[22504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249862, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:37:42.898607 22878 init.cc:377] Logged in from keytab as kudu/127.6.212.3@KRBTEST.COM (short username kudu)
I20260924 11:37:42.900118 22878 webserver.cc:460] Webserver started at http://127.6.212.3:32777/ using document root <none> and password file <none>
I20260924 11:37:42.900799 22878 fs_manager.cc:359] Metadata directory not provided
I20260924 11:37:42.900897 22878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:37:42.901149 22878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:37:42.902534 22738 heartbeater.cc:503] Master 127.6.212.62:36825 was elected leader, sending a full tablet report...
I20260924 11:37:42.904747 22878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-2/data/instance:
uuid: "8832bc897e9d4f2191318fcdba73bb9d"
format_stamp: "Formatted at 2026-09-24 11:37:42 on dist-test-slave-cq7c"
server_key: "d171b7ddb0334a886d3649ce236fb813"
server_key_iv: "115f046321080e9ced9fc85257c19380"
server_key_version: "encryptionkey@0"
I20260924 11:37:42.905454 22878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance:
uuid: "8832bc897e9d4f2191318fcdba73bb9d"
format_stamp: "Formatted at 2026-09-24 11:37:42 on dist-test-slave-cq7c"
server_key: "d171b7ddb0334a886d3649ce236fb813"
server_key_iv: "115f046321080e9ced9fc85257c19380"
server_key_version: "encryptionkey@0"
I20260924 11:37:42.910486 22878 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 11:37:42.914162 22893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:42.915851 22878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 11:37:42.916025 22878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-2/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-2/wal
uuid: "8832bc897e9d4f2191318fcdba73bb9d"
format_stamp: "Formatted at 2026-09-24 11:37:42 on dist-test-slave-cq7c"
server_key: "d171b7ddb0334a886d3649ce236fb813"
server_key_iv: "115f046321080e9ced9fc85257c19380"
server_key_version: "encryptionkey@0"
I20260924 11:37:42.916215 22878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:37:42.930794 22878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:37:42.935655 22878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:37:42.935905 22878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:37:42.936831 22878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:37:42.938153 22878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:37:42.938223 22878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:42.938302 22878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:37:42.938349 22878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:37:42.972563 22878 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.3:46385
I20260924 11:37:42.972611 23005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.3:46385 every 8 connection(s)
I20260924 11:37:42.973965 22878 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
I20260924 11:37:42.977385 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 22878
I20260924 11:37:42.977608 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance
I20260924 11:37:42.978286 6992 external_mini_cluster.cc:1428] Setting key fb5b9df79a1960a2471c63e409459239
Sep 24 11:37:42 dist-test-slave-cq7c krb5kdc[22504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249862, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.3@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:37:42.995819 23008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:42.976865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:38797 (local address 127.6.212.62:36825)
0924 11:37:42.977323 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:42.977327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:42.978647 (+ 1320us) server_negotiation.cc:407] Connection header received
0924 11:37:42.980914 (+ 2267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:42.980920 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:42.981017 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:42.981138 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:42.982827 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:42.983715 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:42.985355 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:42.985795 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:42.985940 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:42.989769 (+ 3829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:42.989775 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:42.989778 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:42.989822 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:42.992649 (+ 2827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:42.993409 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:42.993416 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:42.993419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:42.993554 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:42.993984 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:42.993990 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:42.994003 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:42.994279 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:42.994446 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:42.995168 (+ 722us) server_negotiation.cc:299] Negotiation successful
0924 11:37:42.995418 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":246,"threads_started":1}
I20260924 11:37:42.996850 23009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:42.977580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.3:38797)
0924 11:37:42.978410 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:42.978449 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 11:37:42.980612 (+ 2163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:37:42.981397 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:37:42.981407 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:37:42.981983 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:37:42.982653 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:42.982674 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:42.983883 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:42.983887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:42.985204 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:42.985216 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:42.986037 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:42.986045 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:42.986294 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:42.987013 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:37:42.987036 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:37:42.989472 (+ 2436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:37:42.992813 (+ 3341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:42.992823 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:42.992850 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:42.993284 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:42.993666 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:37:42.993671 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:37:42.993675 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:37:42.993872 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:37:42.994411 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:37:42.994419 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:37:42.994786 (+ 367us) client_negotiation.cc:769] Sending connection context
0924 11:37:42.995079 (+ 293us) client_negotiation.cc:240] Negotiation successful
0924 11:37:42.995671 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":350,"threads_started":1}
I20260924 11:37:42.998165 23006 heartbeater.cc:347] Connected to a master server at 127.6.212.62:36825
I20260924 11:37:42.998453 23006 heartbeater.cc:464] Registering TS with master...
I20260924 11:37:42.999130 23006 heartbeater.cc:511] Master 127.6.212.62:36825 requested a full tablet report, sending...
I20260924 11:37:43.000810 22549 ts_manager.cc:194] Registered new tserver with Master: 8832bc897e9d4f2191318fcdba73bb9d (127.6.212.3:46385)
I20260924 11:37:43.001703 22549 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.3@KRBTEST.COM'} at 127.6.212.3:38797
I20260924 11:37:43.006402 6992 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:37:43.026046 23008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:43.011202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41032 (local address 127.6.212.62:36825)
0924 11:37:43.011438 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:43.011443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:43.011827 (+ 384us) server_negotiation.cc:407] Connection header received
0924 11:37:43.011970 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:43.011974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:43.012055 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:43.012176 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:43.013646 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:43.014574 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:43.016316 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:43.016786 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:43.016896 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:43.019710 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:43.019716 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:43.019719 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:43.019759 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:43.022781 (+ 3022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:43.023727 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:43.023731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:43.023733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:43.023878 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:43.024462 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:43.024467 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:43.024469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:43.024836 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:43.025031 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:43.025400 (+ 369us) server_negotiation.cc:299] Negotiation successful
0924 11:37:43.025653 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 11:37:43.030464 22549 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41032:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:37:43.037951 22549 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:37:43.069643 23019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:43.051772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.2:36853 (local address 127.0.0.1:32810)
0924 11:37:43.052576 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:43.052620 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 11:37:43.052827 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:37:43.054325 (+ 1498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:37:43.054337 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:37:43.054391 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:37:43.054926 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:43.054944 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:43.057763 (+ 2819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:43.057773 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:43.065883 (+ 8110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:43.065914 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:43.068550 (+ 2636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:43.068558 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:43.068791 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:43.068862 (+ 71us) client_negotiation.cc:769] Sending connection context
0924 11:37:43.068982 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 11:37:43.069200 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":344,"threads_started":1}
I20260924 11:37:43.071121 23020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:43.052030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32810 (local address 127.6.212.2:36853)
0924 11:37:43.053451 (+ 1421us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:43.053464 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:43.053503 (+ 39us) server_negotiation.cc:407] Connection header received
0924 11:37:43.053631 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:43.053642 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:43.053977 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:43.055189 (+ 1212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:37:43.055473 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:43.057559 (+ 2086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:43.066262 (+ 8703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:43.068313 (+ 2051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:43.070239 (+ 1926us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:43.070482 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:43.070567 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 11:37:43.070774 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1242,"thread_start_us":1161,"threads_started":1}
I20260924 11:37:43.075731 22807 tablet_service.cc:1467] Processing CreateTablet for tablet e036f83cf2b04caf95f0576d72df13c1 (DEFAULT_TABLE table=test-table [id=68df037eb1bd4578b64cc53b523ee34a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:37:43.076932 23023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:43.052461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.3:46385 (local address 127.0.0.1:46048)
0924 11:37:43.062688 (+ 10227us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:43.062718 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 11:37:43.062912 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:37:43.069502 (+ 6590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:37:43.069508 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:37:43.069535 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:37:43.069922 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:43.069936 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:43.071932 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:43.071939 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:43.073985 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:43.074003 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:43.075975 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:43.075982 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:43.076165 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:43.076200 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 11:37:43.076313 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 11:37:43.076466 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10036,"mutex_wait_us":1,"thread_start_us":15178,"threads_started":1}
I20260924 11:37:43.076956 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:43.052734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46048 (local address 127.6.212.3:46385)
0924 11:37:43.068969 (+ 16235us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:43.068976 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:43.069003 (+ 27us) server_negotiation.cc:407] Connection header received
0924 11:37:43.069083 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:43.069088 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:43.069290 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:43.069520 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:37:43.070159 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:43.071802 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:43.074220 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:43.075773 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:43.075898 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:43.076136 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:43.076376 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 11:37:43.076542 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16092,"thread_start_us":17022,"threads_started":1}
I20260924 11:37:43.077616 23024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:43.053669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38966 (local address 127.6.212.1:46119)
0924 11:37:43.068889 (+ 15220us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:43.068899 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:43.068929 (+ 30us) server_negotiation.cc:407] Connection header received
0924 11:37:43.069039 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:43.069048 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:43.069317 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:43.069519 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:37:43.070840 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:43.072521 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:43.074322 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:43.075761 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:43.075898 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:43.076139 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:43.077048 (+ 909us) server_negotiation.cc:299] Negotiation successful
0924 11:37:43.077248 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15025,"thread_start_us":14836,"threads_started":1}
I20260924 11:37:43.077883 22807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e036f83cf2b04caf95f0576d72df13c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:37:43.078174 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:43.052966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:46119 (local address 127.0.0.1:38966)
0924 11:37:43.068015 (+ 15049us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:43.068043 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 11:37:43.068220 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:37:43.069832 (+ 1612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:37:43.069837 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:37:43.069865 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:37:43.070649 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:43.070664 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:43.072683 (+ 2019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:43.072691 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:43.074131 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:43.074146 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:43.076317 (+ 2171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:43.076323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:43.076505 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:43.076536 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 11:37:43.076634 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 11:37:43.076827 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14862,"mutex_wait_us":1,"thread_start_us":9361,"threads_started":1}
I20260924 11:37:43.081454 22673 tablet_service.cc:1467] Processing CreateTablet for tablet e036f83cf2b04caf95f0576d72df13c1 (DEFAULT_TABLE table=test-table [id=68df037eb1bd4578b64cc53b523ee34a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:37:43.081283 22941 tablet_service.cc:1467] Processing CreateTablet for tablet e036f83cf2b04caf95f0576d72df13c1 (DEFAULT_TABLE table=test-table [id=68df037eb1bd4578b64cc53b523ee34a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:37:43.082687 22673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e036f83cf2b04caf95f0576d72df13c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:37:43.082772 22941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e036f83cf2b04caf95f0576d72df13c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:37:43.093586 23025 tablet_bootstrap.cc:492] T e036f83cf2b04caf95f0576d72df13c1 P 63e7d703fab14977bd19cb32ff984f5e: Bootstrap starting.
I20260924 11:37:43.095629 23026 tablet_bootstrap.cc:492] T e036f83cf2b04caf95f0576d72df13c1 P 62c2c7c8728542e5a9f8f9851794a98d: Bootstrap starting.
I20260924 11:37:43.097599 23027 tablet_bootstrap.cc:492] T e036f83cf2b04caf95f0576d72df13c1 P 8832bc897e9d4f2191318fcdba73bb9d: Bootstrap starting.
I20260924 11:37:43.098104 23025 tablet_bootstrap.cc:654] T e036f83cf2b04caf95f0576d72df13c1 P 63e7d703fab14977bd19cb32ff984f5e: Neither blocks nor log segments found. Creating new log.
I20260924 11:37:43.099138 23026 tablet_bootstrap.cc:654] T e036f83cf2b04caf95f0576d72df13c1 P 62c2c7c8728542e5a9f8f9851794a98d: Neither blocks nor log segments found. Creating new log.
I20260924 11:37:43.099753 23025 log.cc:824] T e036f83cf2b04caf95f0576d72df13c1 P 63e7d703fab14977bd19cb32ff984f5e: Log is configured to *not* fsync() on all Append() calls
I20260924 11:37:43.100271 23026 log.cc:824] T e036f83cf2b04caf95f0576d72df13c1 P 62c2c7c8728542e5a9f8f9851794a98d: Log is configured to *not* fsync() on all Append() calls
I20260924 11:37:43.101125 23027 tablet_bootstrap.cc:654] T e036f83cf2b04caf95f0576d72df13c1 P 8832bc897e9d4f2191318fcdba73bb9d: Neither blocks nor log segments found. Creating new log.
I20260924 11:37:43.102387 23027 log.cc:824] T e036f83cf2b04caf95f0576d72df13c1 P 8832bc897e9d4f2191318fcdba73bb9d: Log is configured to *not* fsync() on all Append() calls
I20260924 11:37:43.103200 23025 tablet_bootstrap.cc:492] T e036f83cf2b04caf95f0576d72df13c1 P 63e7d703fab14977bd19cb32ff984f5e: No bootstrap required, opened a new log
I20260924 11:37:43.103510 23025 ts_tablet_manager.cc:1397] T e036f83cf2b04caf95f0576d72df13c1 P 63e7d703fab14977bd19cb32ff984f5e: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260924 11:37:43.104848 23026 tablet_bootstrap.cc:492] T e036f83cf2b04caf95f0576d72df13c1 P 62c2c7c8728542e5a9f8f9851794a98d: No bootstrap required, opened a new log
I20260924 11:37:43.105123 23026 ts_tablet_manager.cc:1397] T e036f83cf2b04caf95f0576d72df13c1 P 62c2c7c8728542e5a9f8f9851794a98d: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260924 11:37:43.105423 23027 tablet_bootstrap.cc:492] T e036f83cf2b04caf95f0576d72df13c1 P 8832bc897e9d4f2191318fcdba73bb9d: No bootstrap required, opened a new log
I20260924 11:37:43.105660 23027 ts_tablet_manager.cc:1397] T e036f83cf2b04caf95f0576d72df13c1 P 8832bc897e9d4f2191318fcdba73bb9d: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 11:37:43.113119 23026 raft_consensus.cc:348] T e036f83cf2b04caf95f0576d72df13c1 P 62c2c7c8728542e5a9f8f9851794a98d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63e7d703fab14977bd19cb32ff984f5e" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36853 } } peers { permanent_uuid: "8832bc897e9d4f2191318fcdba73bb9d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 46385 } } peers { permanent_uuid: "62c2c7c8728542e5a9f8f9851794a98d" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46119 } }
I20260924 11:37:43.113517 23026 raft_consensus.cc:374] T e036f83cf2b04caf95f0576d72df13c1 P 62c2c7c8728542e5a9f8f9851794a98d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:37:43.113628 23026 raft_consensus.cc:729] T e036f83cf2b04caf95f0576d72df13c1 P 62c2c7c8728542e5a9f8f9851794a98d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62c2c7c8728542e5a9f8f9851794a98d, State: Initialized, Role: FOLLOWER
I20260924 11:37:43.114329 23026 consensus_queue.cc:260] T e036f83cf2b04caf95f0576d72df13c1 P 62c2c7c8728542e5a9f8f9851794a98d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63e7d703fab14977bd19cb32ff984f5e" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36853 } } peers { permanent_uuid: "8832bc897e9d4f2191318fcdba73bb9d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 46385 } } peers { permanent_uuid: "62c2c7c8728542e5a9f8f9851794a98d" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46119 } }
I20260924 11:37:43.114423 23025 raft_consensus.cc:348] T e036f83cf2b04caf95f0576d72df13c1 P 63e7d703fab14977bd19cb32ff984f5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63e7d703fab14977bd19cb32ff984f5e" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36853 } } peers { permanent_uuid: "8832bc897e9d4f2191318fcdba73bb9d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 46385 } } peers { permanent_uuid: "62c2c7c8728542e5a9f8f9851794a98d" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46119 } }
I20260924 11:37:43.114889 23025 raft_consensus.cc:374] T e036f83cf2b04caf95f0576d72df13c1 P 63e7d703fab14977bd19cb32ff984f5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:37:43.114986 23025 raft_consensus.cc:729] T e036f83cf2b04caf95f0576d72df13c1 P 63e7d703fab14977bd19cb32ff984f5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63e7d703fab14977bd19cb32ff984f5e, State: Initialized, Role: FOLLOWER
I20260924 11:37:43.115576 23025 consensus_queue.cc:260] T e036f83cf2b04caf95f0576d72df13c1 P 63e7d703fab14977bd19cb32ff984f5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63e7d703fab14977bd19cb32ff984f5e" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36853 } } peers { permanent_uuid: "8832bc897e9d4f2191318fcdba73bb9d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 46385 } } peers { permanent_uuid: "62c2c7c8728542e5a9f8f9851794a98d" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46119 } }
I20260924 11:37:43.116418 23027 raft_consensus.cc:348] T e036f83cf2b04caf95f0576d72df13c1 P 8832bc897e9d4f2191318fcdba73bb9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63e7d703fab14977bd19cb32ff984f5e" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36853 } } peers { permanent_uuid: "8832bc897e9d4f2191318fcdba73bb9d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 46385 } } peers { permanent_uuid: "62c2c7c8728542e5a9f8f9851794a98d" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46119 } }
I20260924 11:37:43.116796 22872 heartbeater.cc:503] Master 127.6.212.62:36825 was elected leader, sending a full tablet report...
I20260924 11:37:43.116812 23027 raft_consensus.cc:374] T e036f83cf2b04caf95f0576d72df13c1 P 8832bc897e9d4f2191318fcdba73bb9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:37:43.116900 23027 raft_consensus.cc:729] T e036f83cf2b04caf95f0576d72df13c1 P 8832bc897e9d4f2191318fcdba73bb9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8832bc897e9d4f2191318fcdba73bb9d, State: Initialized, Role: FOLLOWER
I20260924 11:37:43.117446 23027 consensus_queue.cc:260] T e036f83cf2b04caf95f0576d72df13c1 P 8832bc897e9d4f2191318fcdba73bb9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63e7d703fab14977bd19cb32ff984f5e" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36853 } } peers { permanent_uuid: "8832bc897e9d4f2191318fcdba73bb9d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 46385 } } peers { permanent_uuid: "62c2c7c8728542e5a9f8f9851794a98d" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46119 } }
I20260924 11:37:43.122786 23026 ts_tablet_manager.cc:1428] T e036f83cf2b04caf95f0576d72df13c1 P 62c2c7c8728542e5a9f8f9851794a98d: Time spent starting tablet: real 0.018s user 0.010s sys 0.007s
I20260924 11:37:43.122779 23025 ts_tablet_manager.cc:1428] T e036f83cf2b04caf95f0576d72df13c1 P 63e7d703fab14977bd19cb32ff984f5e: Time spent starting tablet: real 0.019s user 0.013s sys 0.000s
I20260924 11:37:43.124522 23027 ts_tablet_manager.cc:1428] T e036f83cf2b04caf95f0576d72df13c1 P 8832bc897e9d4f2191318fcdba73bb9d: Time spent starting tablet: real 0.019s user 0.014s sys 0.005s
I20260924 11:37:43.124724 23006 heartbeater.cc:503] Master 127.6.212.62:36825 was elected leader, sending a full tablet report...
W20260924 11:37:43.175696 22873 tablet.cc:2369] T e036f83cf2b04caf95f0576d72df13c1 P 63e7d703fab14977bd19cb32ff984f5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:37:43.178694 23031 raft_consensus.cc:482] T e036f83cf2b04caf95f0576d72df13c1 P 62c2c7c8728542e5a9f8f9851794a98d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:37:43.179003 23031 raft_consensus.cc:504] T e036f83cf2b04caf95f0576d72df13c1 P 62c2c7c8728542e5a9f8f9851794a98d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63e7d703fab14977bd19cb32ff984f5e" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36853 } } peers { permanent_uuid: "8832bc897e9d4f2191318fcdba73bb9d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 46385 } } peers { permanent_uuid: "62c2c7c8728542e5a9f8f9851794a98d" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46119 } }
I20260924 11:37:43.180894 23031 leader_election.cc:288] T e036f83cf2b04caf95f0576d72df13c1 P 62c2c7c8728542e5a9f8f9851794a98d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63e7d703fab14977bd19cb32ff984f5e (127.6.212.2:36853), 8832bc897e9d4f2191318fcdba73bb9d (127.6.212.3:46385)
I20260924 11:37:43.188448 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:43.181403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:46871 (local address 127.6.212.3:46385)
0924 11:37:43.181659 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:43.181665 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:43.182750 (+ 1085us) server_negotiation.cc:407] Connection header received
0924 11:37:43.182910 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:43.182914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:43.183014 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:43.183138 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:37:43.183833 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:43.184544 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:43.186602 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:43.187454 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:43.187555 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:43.187669 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:43.188015 (+ 346us) server_negotiation.cc:299] Negotiation successful
0924 11:37:43.188163 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 11:37:43.188480 23020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:43.181383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:57401 (local address 127.6.212.2:36853)
0924 11:37:43.181603 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:43.181611 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:43.182574 (+ 963us) server_negotiation.cc:407] Connection header received
0924 11:37:43.182777 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:43.182783 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:43.182867 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:43.182962 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:37:43.183755 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:43.184493 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:43.186659 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:43.187537 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:43.187678 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:43.187770 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:43.188079 (+ 309us) server_negotiation.cc:299] Negotiation successful
0924 11:37:43.188210 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 11:37:43.188596 23035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:43.181726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.3:46385 (local address 127.6.212.1:46871)
0924 11:37:43.182614 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:43.182631 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:37:43.182783 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:37:43.183173 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:37:43.183176 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:37:43.183205 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:37:43.183636 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:43.183647 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:43.184782 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:43.184786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:43.186436 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:43.186451 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:43.187671 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:43.187677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:43.187860 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:43.187885 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 11:37:43.187960 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 11:37:43.188115 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"mutex_wait_us":47,"thread_start_us":404,"threads_started":1}
I20260924 11:37:43.188668 23034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:43.181591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.2:36853 (local address 127.6.212.1:57401)
0924 11:37:43.182433 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0924 11:37:43.182449 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:37:43.182626 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:37:43.182987 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:37:43.182993 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:37:43.183019 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:37:43.183595 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:43.183610 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:43.184696 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:43.184705 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:43.186466 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:37:43.186477 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:37:43.187771 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:37:43.187776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:37:43.187899 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:43.187921 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 11:37:43.188031 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 11:37:43.188160 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"mutex_wait_us":86,"spinlock_wait_cycles":24448,"thread_start_us":496,"threads_started":1}
I20260924 11:37:43.189790 22827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e036f83cf2b04caf95f0576d72df13c1" candidate_uuid: "62c2c7c8728542e5a9f8f9851794a98d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63e7d703fab14977bd19cb32ff984f5e" is_pre_election: true
I20260924 11:37:43.190042 22961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e036f83cf2b04caf95f0576d72df13c1" candidate_uuid: "62c2c7c8728542e5a9f8f9851794a98d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8832bc897e9d4f2191318fcdba73bb9d" is_pre_election: true
I20260924 11:37:43.190346 22827 raft_consensus.cc:2454] T e036f83cf2b04caf95f0576d72df13c1 P 63e7d703fab14977bd19cb32ff984f5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62c2c7c8728542e5a9f8f9851794a98d in term 0.
I20260924 11:37:43.190600 22961 raft_consensus.cc:2454] T e036f83cf2b04caf95f0576d72df13c1 P 8832bc897e9d4f2191318fcdba73bb9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62c2c7c8728542e5a9f8f9851794a98d in term 0.
I20260924 11:37:43.191129 22629 leader_election.cc:302] T e036f83cf2b04caf95f0576d72df13c1 P 62c2c7c8728542e5a9f8f9851794a98d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62c2c7c8728542e5a9f8f9851794a98d, 63e7d703fab14977bd19cb32ff984f5e; no voters:
I20260924 11:37:43.191658 23031 raft_consensus.cc:2781] T e036f83cf2b04caf95f0576d72df13c1 P 62c2c7c8728542e5a9f8f9851794a98d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:37:43.191825 23031 raft_consensus.cc:482] T e036f83cf2b04caf95f0576d72df13c1 P 62c2c7c8728542e5a9f8f9851794a98d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:37:43.191956 23031 raft_consensus.cc:3037] T e036f83cf2b04caf95f0576d72df13c1 P 62c2c7c8728542e5a9f8f9851794a98d [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:37:43.194224 23031 raft_consensus.cc:504] T e036f83cf2b04caf95f0576d72df13c1 P 62c2c7c8728542e5a9f8f9851794a98d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63e7d703fab14977bd19cb32ff984f5e" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36853 } } peers { permanent_uuid: "8832bc897e9d4f2191318fcdba73bb9d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 46385 } } peers { permanent_uuid: "62c2c7c8728542e5a9f8f9851794a98d" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46119 } }
I20260924 11:37:43.194793 23031 leader_election.cc:288] T e036f83cf2b04caf95f0576d72df13c1 P 62c2c7c8728542e5a9f8f9851794a98d [CANDIDATE]: Term 1 election: Requested vote from peers 63e7d703fab14977bd19cb32ff984f5e (127.6.212.2:36853), 8832bc897e9d4f2191318fcdba73bb9d (127.6.212.3:46385)
I20260924 11:37:43.195331 22827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e036f83cf2b04caf95f0576d72df13c1" candidate_uuid: "62c2c7c8728542e5a9f8f9851794a98d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63e7d703fab14977bd19cb32ff984f5e"
I20260924 11:37:43.195333 22961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e036f83cf2b04caf95f0576d72df13c1" candidate_uuid: "62c2c7c8728542e5a9f8f9851794a98d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8832bc897e9d4f2191318fcdba73bb9d"
I20260924 11:37:43.195569 22827 raft_consensus.cc:3037] T e036f83cf2b04caf95f0576d72df13c1 P 63e7d703fab14977bd19cb32ff984f5e [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:37:43.195649 22961 raft_consensus.cc:3037] T e036f83cf2b04caf95f0576d72df13c1 P 8832bc897e9d4f2191318fcdba73bb9d [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:37:43.198829 22961 raft_consensus.cc:2454] T e036f83cf2b04caf95f0576d72df13c1 P 8832bc897e9d4f2191318fcdba73bb9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62c2c7c8728542e5a9f8f9851794a98d in term 1.
I20260924 11:37:43.198829 22827 raft_consensus.cc:2454] T e036f83cf2b04caf95f0576d72df13c1 P 63e7d703fab14977bd19cb32ff984f5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62c2c7c8728542e5a9f8f9851794a98d in term 1.
I20260924 11:37:43.199280 22629 leader_election.cc:302] T e036f83cf2b04caf95f0576d72df13c1 P 62c2c7c8728542e5a9f8f9851794a98d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62c2c7c8728542e5a9f8f9851794a98d, 63e7d703fab14977bd19cb32ff984f5e; no voters:
I20260924 11:37:43.199643 23031 raft_consensus.cc:2781] T e036f83cf2b04caf95f0576d72df13c1 P 62c2c7c8728542e5a9f8f9851794a98d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:37:43.200153 23031 raft_consensus.cc:686] T e036f83cf2b04caf95f0576d72df13c1 P 62c2c7c8728542e5a9f8f9851794a98d [term 1 LEADER]: Becoming Leader. State: Replica: 62c2c7c8728542e5a9f8f9851794a98d, State: Running, Role: LEADER
I20260924 11:37:43.200752 23031 consensus_queue.cc:237] T e036f83cf2b04caf95f0576d72df13c1 P 62c2c7c8728542e5a9f8f9851794a98d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63e7d703fab14977bd19cb32ff984f5e" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36853 } } peers { permanent_uuid: "8832bc897e9d4f2191318fcdba73bb9d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 46385 } } peers { permanent_uuid: "62c2c7c8728542e5a9f8f9851794a98d" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46119 } }
I20260924 11:37:43.205919 22549 catalog_manager.cc:5495] T e036f83cf2b04caf95f0576d72df13c1 P 62c2c7c8728542e5a9f8f9851794a98d reported cstate change: term changed from 0 to 1, leader changed from <none> to 62c2c7c8728542e5a9f8f9851794a98d (127.6.212.1). New cstate: current_term: 1 leader_uuid: "62c2c7c8728542e5a9f8f9851794a98d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63e7d703fab14977bd19cb32ff984f5e" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8832bc897e9d4f2191318fcdba73bb9d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 46385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "62c2c7c8728542e5a9f8f9851794a98d" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46119 } health_report { overall_health: HEALTHY } } }
W20260924 11:37:43.225800 23007 tablet.cc:2369] T e036f83cf2b04caf95f0576d72df13c1 P 8832bc897e9d4f2191318fcdba73bb9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:37:43.236580 23024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:43.230878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38978 (local address 127.6.212.1:46119)
0924 11:37:43.231133 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:43.231151 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:43.231167 (+ 16us) server_negotiation.cc:407] Connection header received
0924 11:37:43.231260 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:43.231264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:43.231339 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:43.231564 (+ 225us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:37:43.231931 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:43.232686 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:43.234260 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:43.234683 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:43.234828 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:43.235344 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:37:43.235983 (+ 639us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:37:43.236142 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:43.236203 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 11:37:43.236288 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 11:37:43.245527 22961 raft_consensus.cc:1261] T e036f83cf2b04caf95f0576d72df13c1 P 8832bc897e9d4f2191318fcdba73bb9d [term 1 FOLLOWER]: Refusing update from remote peer 62c2c7c8728542e5a9f8f9851794a98d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:37:43.245527 22827 raft_consensus.cc:1261] T e036f83cf2b04caf95f0576d72df13c1 P 63e7d703fab14977bd19cb32ff984f5e [term 1 FOLLOWER]: Refusing update from remote peer 62c2c7c8728542e5a9f8f9851794a98d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:37:43.246833 23036 consensus_queue.cc:1035] T e036f83cf2b04caf95f0576d72df13c1 P 62c2c7c8728542e5a9f8f9851794a98d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63e7d703fab14977bd19cb32ff984f5e" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:37:43.247277 23031 consensus_queue.cc:1035] T e036f83cf2b04caf95f0576d72df13c1 P 62c2c7c8728542e5a9f8f9851794a98d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8832bc897e9d4f2191318fcdba73bb9d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 46385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:37:43.260882 23039 mvcc.cc:205] Tried to move back new op lower bound from 7332863439841673216 to 7332863439674630144. Current Snapshot: MvccSnapshot[applied={T|T < 7332863439841673216}]
I20260924 11:37:43.262251 23040 mvcc.cc:205] Tried to move back new op lower bound from 7332863439841673216 to 7332863439674630144. Current Snapshot: MvccSnapshot[applied={T|T < 7332863439841673216}]
I20260924 11:37:43.273720 22549 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:37:43.274026 22549 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:37:43.277760 22549 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf: Sending DeleteTablet for 3 replicas of tablet e036f83cf2b04caf95f0576d72df13c1
I20260924 11:37:43.279069 22941 tablet_service.cc:1514] Processing DeleteTablet for tablet e036f83cf2b04caf95f0576d72df13c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:37:43 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:46048
I20260924 11:37:43.279112 22673 tablet_service.cc:1514] Processing DeleteTablet for tablet e036f83cf2b04caf95f0576d72df13c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:37:43 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:38966
I20260924 11:37:43.279112 22807 tablet_service.cc:1514] Processing DeleteTablet for tablet e036f83cf2b04caf95f0576d72df13c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:37:43 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:32810
I20260924 11:37:43.280480 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 22610
I20260924 11:37:43.290958 23048 tablet_replica.cc:330] T e036f83cf2b04caf95f0576d72df13c1 P 8832bc897e9d4f2191318fcdba73bb9d: stopping tablet replica
I20260924 11:37:43.291682 23048 raft_consensus.cc:2229] T e036f83cf2b04caf95f0576d72df13c1 P 8832bc897e9d4f2191318fcdba73bb9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:37:43.292366 23048 raft_consensus.cc:2258] T e036f83cf2b04caf95f0576d72df13c1 P 8832bc897e9d4f2191318fcdba73bb9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:37:43.293536 23049 tablet_replica.cc:330] T e036f83cf2b04caf95f0576d72df13c1 P 63e7d703fab14977bd19cb32ff984f5e: stopping tablet replica
I20260924 11:37:43.295132 23048 ts_tablet_manager.cc:1905] T e036f83cf2b04caf95f0576d72df13c1 P 8832bc897e9d4f2191318fcdba73bb9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:37:43.302476 23048 ts_tablet_manager.cc:1918] T e036f83cf2b04caf95f0576d72df13c1 P 8832bc897e9d4f2191318fcdba73bb9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:37:43.303239 23050 tablet_replica.cc:330] T e036f83cf2b04caf95f0576d72df13c1 P 62c2c7c8728542e5a9f8f9851794a98d: stopping tablet replica
I20260924 11:37:43.303929 23049 raft_consensus.cc:2229] T e036f83cf2b04caf95f0576d72df13c1 P 63e7d703fab14977bd19cb32ff984f5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:37:43.304095 23050 raft_consensus.cc:2229] T e036f83cf2b04caf95f0576d72df13c1 P 62c2c7c8728542e5a9f8f9851794a98d [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:37:43.304523 23049 raft_consensus.cc:2258] T e036f83cf2b04caf95f0576d72df13c1 P 63e7d703fab14977bd19cb32ff984f5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:37:43.304924 23050 raft_consensus.cc:2258] T e036f83cf2b04caf95f0576d72df13c1 P 62c2c7c8728542e5a9f8f9851794a98d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:37:43.305006 23048 log.cc:1196] T e036f83cf2b04caf95f0576d72df13c1 P 8832bc897e9d4f2191318fcdba73bb9d: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-2/wal/wals/e036f83cf2b04caf95f0576d72df13c1
I20260924 11:37:43.305627 23048 ts_tablet_manager.cc:1939] T e036f83cf2b04caf95f0576d72df13c1 P 8832bc897e9d4f2191318fcdba73bb9d: Deleting consensus metadata
I20260924 11:37:43.306874 23050 ts_tablet_manager.cc:1905] T e036f83cf2b04caf95f0576d72df13c1 P 62c2c7c8728542e5a9f8f9851794a98d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:37:43.306874 23049 ts_tablet_manager.cc:1905] T e036f83cf2b04caf95f0576d72df13c1 P 63e7d703fab14977bd19cb32ff984f5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:37:43.307767 22535 catalog_manager.cc:4841] TS 8832bc897e9d4f2191318fcdba73bb9d (127.6.212.3:46385): tablet e036f83cf2b04caf95f0576d72df13c1 (table test-table [id=68df037eb1bd4578b64cc53b523ee34a]) successfully deleted
Sep 24 11:37:43 dist-test-slave-cq7c krb5kdc[22504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.1@KRBTEST.COM
I20260924 11:37:43.313941 23049 ts_tablet_manager.cc:1918] T e036f83cf2b04caf95f0576d72df13c1 P 63e7d703fab14977bd19cb32ff984f5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:37:43.314126 23049 log.cc:1196] T e036f83cf2b04caf95f0576d72df13c1 P 63e7d703fab14977bd19cb32ff984f5e: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-1/wal/wals/e036f83cf2b04caf95f0576d72df13c1
I20260924 11:37:43.314642 23049 ts_tablet_manager.cc:1939] T e036f83cf2b04caf95f0576d72df13c1 P 63e7d703fab14977bd19cb32ff984f5e: Deleting consensus metadata
I20260924 11:37:43.316543 22536 catalog_manager.cc:4841] TS 63e7d703fab14977bd19cb32ff984f5e (127.6.212.2:36853): tablet e036f83cf2b04caf95f0576d72df13c1 (table test-table [id=68df037eb1bd4578b64cc53b523ee34a]) successfully deleted
I20260924 11:37:43.318168 23024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:43.281902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38992 (local address 127.6.212.1:46119)
0924 11:37:43.282135 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:43.282144 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:43.289316 (+ 7172us) server_negotiation.cc:407] Connection header received
0924 11:37:43.289538 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:43.289548 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:43.289694 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:43.289846 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:43.295982 (+ 6136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:43.297723 (+ 1741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:43.301182 (+ 3459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:43.302260 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:43.302410 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:43.309537 (+ 7127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:43.309551 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:43.309579 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:43.309641 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:43.313924 (+ 4283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:43.315337 (+ 1413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:43.315353 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:43.315398 (+ 45us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:43.315598 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:43.316114 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:43.316120 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:43.316123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:43.316843 (+ 720us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:43.317224 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:43.317533 (+ 309us) server_negotiation.cc:299] Negotiation successful
0924 11:37:43.317795 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 11:37:43.319182 22733 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:37:43.419422 23050 ts_tablet_manager.cc:1918] T e036f83cf2b04caf95f0576d72df13c1 P 62c2c7c8728542e5a9f8f9851794a98d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:37:43.419730 23050 log.cc:1196] T e036f83cf2b04caf95f0576d72df13c1 P 62c2c7c8728542e5a9f8f9851794a98d: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-0/wal/wals/e036f83cf2b04caf95f0576d72df13c1
I20260924 11:37:43.420426 23050 ts_tablet_manager.cc:1939] T e036f83cf2b04caf95f0576d72df13c1 P 62c2c7c8728542e5a9f8f9851794a98d: Deleting consensus metadata
I20260924 11:37:43.422365 22536 catalog_manager.cc:4841] TS 62c2c7c8728542e5a9f8f9851794a98d (127.6.212.1:46119): tablet e036f83cf2b04caf95f0576d72df13c1 (table test-table [id=68df037eb1bd4578b64cc53b523ee34a]) successfully deleted
I20260924 11:37:43.422452 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 22610
I20260924 11:37:43.444783 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 22744
Sep 24 11:37:43 dist-test-slave-cq7c krb5kdc[22504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.2@KRBTEST.COM
I20260924 11:37:43.461344 23020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:43.445693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32820 (local address 127.6.212.2:36853)
0924 11:37:43.445898 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:43.445904 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:43.445925 (+ 21us) server_negotiation.cc:407] Connection header received
0924 11:37:43.446047 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:43.446089 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:43.446173 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:43.446289 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:43.447682 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:43.448465 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:43.449917 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:43.450338 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:43.450485 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:43.455441 (+ 4956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:43.455459 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:43.455477 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:43.455521 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:43.458439 (+ 2918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:43.459164 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:43.459172 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:43.459174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:43.459306 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:43.459790 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:43.459793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:43.459795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:43.460214 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:43.460450 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:43.460753 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 11:37:43.460930 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 11:37:43.462148 22867 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:37:43.554544 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 22744
I20260924 11:37:43.572223 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 22878
Sep 24 11:37:43 dist-test-slave-cq7c krb5kdc[22504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.3@KRBTEST.COM
I20260924 11:37:43.588776 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:37:43.573108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46056 (local address 127.6.212.3:46385)
0924 11:37:43.573277 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 11:37:43.573281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:37:43.573300 (+ 19us) server_negotiation.cc:407] Connection header received
0924 11:37:43.573456 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:37:43.573460 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:37:43.573527 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:37:43.573616 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:37:43.575160 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:43.576075 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:43.577603 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:37:43.578062 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:37:43.578214 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:37:43.582897 (+ 4683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:37:43.582913 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:37:43.582930 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:37:43.582974 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:37:43.585747 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:43.586497 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:43.586505 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:43.586507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:43.586665 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:37:43.587058 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:37:43.587064 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:37:43.587067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:37:43.587534 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:37:43.587829 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0924 11:37:43.588179 (+ 350us) server_negotiation.cc:299] Negotiation successful
0924 11:37:43.588409 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 11:37:43.589479 23001 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:37:43.681404 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 22878
I20260924 11:37:43.699800 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 22517
I20260924 11:37:43.701448 22579 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:37:43.823804 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 22517
I20260924 11:38:03.848691 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:36825
--webserver_interface=127.6.212.62
--webserver_port=36195
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:36245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:36825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:38:04.199643 23056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:38:04.200022 23056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:38:04.200218 23056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:38:04.207687 23056 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 11:38:04.207788 23056 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 11:38:04.207826 23056 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 11:38:04.207923 23056 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:38:04.208031 23056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:38:04.208149 23056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:38:04.208205 23056 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:38:04.208298 23056 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:38:04.218581 23056 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:36245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:36825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:36825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.212.62
--webserver_port=36195
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:38:04.219959 23056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:38:04.221338 23056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:38:04.247273 23065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:38:04.247257 23063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:38:04.247470 23062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:38:04.247759 23056 server_base.cc:1034] running on GCE node
I20260924 11:38:04.249058 23056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:38:04.250756 23056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:38:04.252020 23056 hybrid_clock.cc:648] HybridClock initialized: now 1790249884251927 us; error 68 us; skew 500 ppm
I20260924 11:38:04.252436 23056 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:38:04 dist-test-slave-cq7c krb5kdc[22504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249884, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:38:04.256368 23056 init.cc:377] Logged in from keytab as kudu/127.6.212.62@KRBTEST.COM (short username kudu)
I20260924 11:38:04.258217 23056 webserver.cc:460] Webserver started at http://127.6.212.62:36195/ using document root <none> and password file <none>
I20260924 11:38:04.259094 23056 fs_manager.cc:359] Metadata directory not provided
I20260924 11:38:04.259256 23056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:38:04.266280 23056 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 11:38:04.269892 23072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:04.271700 23056 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 11:38:04.271917 23056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "ac21188da6be46928a99557e485926bf"
format_stamp: "Formatted at 2026-09-24 11:37:41 on dist-test-slave-cq7c"
server_key: "c9a674c6ddf3b9c9bfe32ff0b221b7c1"
server_key_iv: "5aeadd200445eafd60a4ecce6a1dfaa2"
server_key_version: "encryptionkey@0"
I20260924 11:38:04.273110 23056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:38:04.288715 23056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:38:04.293154 23056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:38:04.293497 23056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:38:04.314042 23056 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.62:36825
I20260924 11:38:04.314086 23123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.62:36825 every 8 connection(s)
I20260924 11:38:04.315860 23056 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:38:04.323488 23124 sys_catalog.cc:263] Verifying existing consensus state
I20260924 11:38:04.325814 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 23056
I20260924 11:38:04.326653 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.1:46119
--local_ip_for_outbound_sockets=127.6.212.1
--tserver_master_addrs=127.6.212.62:36825
--webserver_port=41729
--webserver_interface=127.6.212.1
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab
--principal=kudu/127.6.212.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:36245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:38:04.327030 23124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf: Bootstrap starting.
I20260924 11:38:04.348188 23124 log.cc:824] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf: Log is configured to *not* fsync() on all Append() calls
I20260924 11:38:04.360944 23124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 11:38:04.361677 23124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf: Bootstrap complete.
I20260924 11:38:04.371630 23124 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac21188da6be46928a99557e485926bf" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36825 } }
I20260924 11:38:04.372538 23124 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac21188da6be46928a99557e485926bf, State: Initialized, Role: FOLLOWER
I20260924 11:38:04.373226 23124 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac21188da6be46928a99557e485926bf" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36825 } }
I20260924 11:38:04.373479 23124 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:38:04.373627 23124 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:38:04.373811 23124 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [term 1 FOLLOWER]: Advancing to term 2
I20260924 11:38:04.378237 23124 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac21188da6be46928a99557e485926bf" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36825 } }
I20260924 11:38:04.378859 23124 leader_election.cc:302] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac21188da6be46928a99557e485926bf; no voters:
I20260924 11:38:04.379702 23124 leader_election.cc:288] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 11:38:04.379853 23128 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [term 2 FOLLOWER]: Leader election won for term 2
I20260924 11:38:04.381289 23128 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [term 2 LEADER]: Becoming Leader. State: Replica: ac21188da6be46928a99557e485926bf, State: Running, Role: LEADER
I20260924 11:38:04.381922 23128 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac21188da6be46928a99557e485926bf" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36825 } }
I20260924 11:38:04.382424 23124 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:38:04.384716 23130 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac21188da6be46928a99557e485926bf. Latest consensus state: current_term: 2 leader_uuid: "ac21188da6be46928a99557e485926bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac21188da6be46928a99557e485926bf" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36825 } } }
I20260924 11:38:04.385102 23130 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [sys.catalog]: This master's current role is: LEADER
I20260924 11:38:04.384878 23129 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ac21188da6be46928a99557e485926bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac21188da6be46928a99557e485926bf" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36825 } } }
I20260924 11:38:04.385308 23129 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [sys.catalog]: This master's current role is: LEADER
I20260924 11:38:04.400807 23135 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:38:04.409273 23135 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:38:04.410774 23135 catalog_manager.cc:1259] Loaded cluster ID: 0b046aa47f3c42a38c8a9b1a5dcd2232
I20260924 11:38:04.410902 23135 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:38:04.415072 23135 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:38:04.415992 23145 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 11:38:04.416994 23135 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf: Loaded TSK: 0
I20260924 11:38:04.457396 23135 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf: Generated new TSK 1
I20260924 11:38:04.457827 23135 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 11:38:04.711440 23126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:38:04.711747 23126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:38:04.711892 23126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:38:04.719514 23126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:38:04.719640 23126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:38:04.719827 23126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.1
I20260924 11:38:04.730100 23126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:36245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab
--principal=kudu/127.6.212.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.1:46119
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.212.1
--webserver_port=41729
--tserver_master_addrs=127.6.212.62:36825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.1
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:38:04.731705 23126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:38:04.733192 23126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:38:04.747701 23151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:38:04.747727 23152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:38:04.748083 23154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:38:04.748997 23126 server_base.cc:1034] running on GCE node
I20260924 11:38:04.749701 23126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:38:04.750869 23126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:38:04.752096 23126 hybrid_clock.cc:648] HybridClock initialized: now 1790249884752044 us; error 52 us; skew 500 ppm
I20260924 11:38:04.752468 23126 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:38:04 dist-test-slave-cq7c krb5kdc[22504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249884, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:38:04.755883 23126 init.cc:377] Logged in from keytab as kudu/127.6.212.1@KRBTEST.COM (short username kudu)
I20260924 11:38:04.757619 23126 webserver.cc:460] Webserver started at http://127.6.212.1:41729/ using document root <none> and password file <none>
I20260924 11:38:04.758363 23126 fs_manager.cc:359] Metadata directory not provided
I20260924 11:38:04.758535 23126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:38:04.764689 23126 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260924 11:38:04.767941 23161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:04.769840 23126 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260924 11:38:04.770087 23126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-0/wal
uuid: "62c2c7c8728542e5a9f8f9851794a98d"
format_stamp: "Formatted at 2026-09-24 11:37:41 on dist-test-slave-cq7c"
server_key: "27ba1496484442fc22dfffff75b3fcc8"
server_key_iv: "95acc6f22192c558b523389e9c3e5bab"
server_key_version: "encryptionkey@0"
I20260924 11:38:04.771150 23126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:38:04.789366 23126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:38:04.793853 23126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:38:04.794166 23126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:38:04.795327 23126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:38:04.797024 23126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:38:04.797152 23126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:04.797298 23126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:38:04.797426 23126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:04.833046 23126 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.1:46119
I20260924 11:38:04.833097 23273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.1:46119 every 8 connection(s)
I20260924 11:38:04.834631 23126 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
I20260924 11:38:04.843808 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 23126
I20260924 11:38:04.844501 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.2:36853
--local_ip_for_outbound_sockets=127.6.212.2
--tserver_master_addrs=127.6.212.62:36825
--webserver_port=41293
--webserver_interface=127.6.212.2
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab
--principal=kudu/127.6.212.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:36245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 11:38:04 dist-test-slave-cq7c krb5kdc[22504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249884, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:38:04.861800 23276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:04.838011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:41915 (local address 127.6.212.62:36825)
0924 11:38:04.838702 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:04.838714 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:04.839494 (+ 780us) server_negotiation.cc:407] Connection header received
0924 11:38:04.842106 (+ 2612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:04.842120 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:04.842538 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:04.842920 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:38:04.844895 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:04.846355 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:04.848223 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:04.849052 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:04.849307 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:04.853509 (+ 4202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:38:04.853518 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:38:04.853539 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:38:04.853588 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:38:04.856937 (+ 3349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:04.857777 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:04.857784 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:04.857791 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:04.857962 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:04.858447 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:04.858454 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:04.858462 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:04.858976 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:38:04.859192 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:04.860079 (+ 887us) server_negotiation.cc:299] Negotiation successful
0924 11:38:04.860474 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":366,"threads_started":1}
I20260924 11:38:04.861794 23277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:04.838138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.1:41915)
0924 11:38:04.839201 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:04.839263 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 11:38:04.841492 (+ 2229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:38:04.843061 (+ 1569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:38:04.843069 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:38:04.843847 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:38:04.844608 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:04.844658 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:04.846561 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:04.846566 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:04.848036 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:04.848049 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:04.849346 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:04.849355 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:04.849664 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:04.850464 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:38:04.850489 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:38:04.853130 (+ 2641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:38:04.857111 (+ 3981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:38:04.857121 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:38:04.857138 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:38:04.857607 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:38:04.858084 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:38:04.858090 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:38:04.858094 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:38:04.858306 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:38:04.859119 (+ 813us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:38:04.859127 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:38:04.859558 (+ 431us) client_negotiation.cc:769] Sending connection context
0924 11:38:04.859824 (+ 266us) client_negotiation.cc:240] Negotiation successful
0924 11:38:04.860437 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"thread_start_us":408,"threads_started":1}
I20260924 11:38:04.865577 23274 heartbeater.cc:347] Connected to a master server at 127.6.212.62:36825
I20260924 11:38:04.866016 23274 heartbeater.cc:464] Registering TS with master...
I20260924 11:38:04.867201 23274 heartbeater.cc:511] Master 127.6.212.62:36825 requested a full tablet report, sending...
I20260924 11:38:04.870565 23089 ts_manager.cc:194] Registered new tserver with Master: 62c2c7c8728542e5a9f8f9851794a98d (127.6.212.1:46119)
I20260924 11:38:04.873420 23089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.1@KRBTEST.COM'} at 127.6.212.1:41915
W20260924 11:38:05.230135 23278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:38:05.230484 23278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:38:05.230589 23278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:38:05.238972 23278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:38:05.239114 23278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:38:05.239246 23278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.2
I20260924 11:38:05.249751 23278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:36245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab
--principal=kudu/127.6.212.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.2:36853
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.212.2
--webserver_port=41293
--tserver_master_addrs=127.6.212.62:36825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.2
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:38:05.251055 23278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:38:05.252492 23278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:38:05.266429 23284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:38:05.266474 23283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:38:05.266497 23286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:38:05.268199 23278 server_base.cc:1034] running on GCE node
I20260924 11:38:05.269037 23278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:38:05.270150 23278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:38:05.271495 23278 hybrid_clock.cc:648] HybridClock initialized: now 1790249885271424 us; error 64 us; skew 500 ppm
I20260924 11:38:05.271871 23278 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:38:05 dist-test-slave-cq7c krb5kdc[22504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249885, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:38:05.275584 23278 init.cc:377] Logged in from keytab as kudu/127.6.212.2@KRBTEST.COM (short username kudu)
I20260924 11:38:05.277379 23278 webserver.cc:460] Webserver started at http://127.6.212.2:41293/ using document root <none> and password file <none>
I20260924 11:38:05.278122 23278 fs_manager.cc:359] Metadata directory not provided
I20260924 11:38:05.278306 23278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:38:05.284408 23278 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260924 11:38:05.287189 23293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:05.288640 23278 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 11:38:05.288861 23278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-1/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-1/wal
uuid: "63e7d703fab14977bd19cb32ff984f5e"
format_stamp: "Formatted at 2026-09-24 11:37:42 on dist-test-slave-cq7c"
server_key: "e97b9476f1007d167819ee0e85b6c5a6"
server_key_iv: "ffb18584d4aa4d09a8742220718447aa"
server_key_version: "encryptionkey@0"
I20260924 11:38:05.289893 23278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:38:05.315536 23278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:38:05.320060 23278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:38:05.320356 23278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:38:05.321426 23278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:38:05.322894 23278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:38:05.323025 23278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:05.323156 23278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:38:05.323243 23278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:05.357548 23278 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.2:36853
I20260924 11:38:05.357637 23405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.2:36853 every 8 connection(s)
I20260924 11:38:05.359108 23278 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
I20260924 11:38:05.363675 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 23278
I20260924 11:38:05.364297 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.3:46385
--local_ip_for_outbound_sockets=127.6.212.3
--tserver_master_addrs=127.6.212.62:36825
--webserver_port=32777
--webserver_interface=127.6.212.3
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab
--principal=kudu/127.6.212.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:36245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 11:38:05 dist-test-slave-cq7c krb5kdc[22504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249885, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.2@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:38:05.383662 23276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:05.362052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:33055 (local address 127.6.212.62:36825)
0924 11:38:05.362304 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:05.362310 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:05.363485 (+ 1175us) server_negotiation.cc:407] Connection header received
0924 11:38:05.365521 (+ 2036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:05.365530 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:05.365643 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:05.365770 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:38:05.367235 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:05.368371 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:05.370997 (+ 2626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:05.371677 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:05.371807 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:05.376383 (+ 4576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:38:05.376390 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:38:05.376393 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:38:05.376430 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:38:05.379893 (+ 3463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:05.380736 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:05.380744 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:05.380750 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:05.380913 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:05.381347 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:05.381355 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:05.381361 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:05.381779 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:38:05.381927 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:05.382829 (+ 902us) server_negotiation.cc:299] Negotiation successful
0924 11:38:05.383146 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 11:38:05.384507 23408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:05.362449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.2:33055)
0924 11:38:05.363290 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:05.363342 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 11:38:05.365232 (+ 1890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:38:05.365994 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:38:05.366002 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:38:05.366498 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:38:05.367057 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:05.367073 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:05.368570 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:05.368577 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:05.370835 (+ 2258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:05.370853 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:05.371915 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:05.371924 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:05.372265 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:05.373151 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:38:05.373182 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:38:05.375999 (+ 2817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:38:05.380070 (+ 4071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:38:05.380078 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:38:05.380094 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:38:05.380601 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:38:05.381033 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:38:05.381037 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:38:05.381039 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:38:05.381229 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:38:05.381929 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:38:05.381935 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:38:05.382405 (+ 470us) client_negotiation.cc:769] Sending connection context
0924 11:38:05.382703 (+ 298us) client_negotiation.cc:240] Negotiation successful
0924 11:38:05.383188 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":343,"threads_started":1}
I20260924 11:38:05.386435 23406 heartbeater.cc:347] Connected to a master server at 127.6.212.62:36825
I20260924 11:38:05.386806 23406 heartbeater.cc:464] Registering TS with master...
I20260924 11:38:05.387730 23406 heartbeater.cc:511] Master 127.6.212.62:36825 requested a full tablet report, sending...
I20260924 11:38:05.389488 23089 ts_manager.cc:194] Registered new tserver with Master: 63e7d703fab14977bd19cb32ff984f5e (127.6.212.2:36853)
I20260924 11:38:05.390892 23089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.2@KRBTEST.COM'} at 127.6.212.2:33055
W20260924 11:38:05.717376 23409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:38:05.717681 23409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:38:05.717778 23409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:38:05.725426 23409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:38:05.725557 23409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:38:05.725675 23409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.3
I20260924 11:38:05.736174 23409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:36245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab
--principal=kudu/127.6.212.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.3:46385
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.212.3
--webserver_port=32777
--tserver_master_addrs=127.6.212.62:36825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.3
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:38:05.737534 23409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:38:05.738811 23409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:38:05.753158 23414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:38:05.753293 23417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:38:05.753381 23415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:38:05.754532 23409 server_base.cc:1034] running on GCE node
I20260924 11:38:05.755281 23409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:38:05.756399 23409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:38:05.757645 23409 hybrid_clock.cc:648] HybridClock initialized: now 1790249885757574 us; error 61 us; skew 500 ppm
I20260924 11:38:05.758006 23409 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:38:05 dist-test-slave-cq7c krb5kdc[22504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249885, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:38:05.765064 23409 init.cc:377] Logged in from keytab as kudu/127.6.212.3@KRBTEST.COM (short username kudu)
I20260924 11:38:05.766935 23409 webserver.cc:460] Webserver started at http://127.6.212.3:32777/ using document root <none> and password file <none>
I20260924 11:38:05.767813 23409 fs_manager.cc:359] Metadata directory not provided
I20260924 11:38:05.768002 23409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:38:05.774199 23409 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 11:38:05.778052 23424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:05.779896 23409 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260924 11:38:05.780112 23409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-2/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-2/wal
uuid: "8832bc897e9d4f2191318fcdba73bb9d"
format_stamp: "Formatted at 2026-09-24 11:37:42 on dist-test-slave-cq7c"
server_key: "d171b7ddb0334a886d3649ce236fb813"
server_key_iv: "115f046321080e9ced9fc85257c19380"
server_key_version: "encryptionkey@0"
I20260924 11:38:05.781188 23409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:38:05.817312 23409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:38:05.821695 23409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:38:05.821990 23409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:38:05.823060 23409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:38:05.824775 23409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:38:05.824916 23409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:05.825062 23409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:38:05.825160 23409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:05.861994 23409 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.3:46385
I20260924 11:38:05.862059 23536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.3:46385 every 8 connection(s)
I20260924 11:38:05.863615 23409 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
I20260924 11:38:05.864846 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 23409
I20260924 11:38:05.878036 23274 heartbeater.cc:503] Master 127.6.212.62:36825 was elected leader, sending a full tablet report...
Sep 24 11:38:05 dist-test-slave-cq7c krb5kdc[22504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249885, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.3@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:38:05.883956 23276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:05.865926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36758 (local address 127.6.212.62:36825)
0924 11:38:05.866118 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:05.866145 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:05.866689 (+ 544us) server_negotiation.cc:407] Connection header received
0924 11:38:05.866951 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:05.866960 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:05.867056 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:05.867183 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:38:05.869681 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:05.870897 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:05.872628 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:05.873235 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:05.873402 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:05.876787 (+ 3385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:38:05.876793 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:38:05.876798 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:38:05.876852 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:38:05.880963 (+ 4111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:05.881736 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:05.881742 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:05.881746 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:05.881881 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:05.882319 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:05.882324 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:05.882327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:05.882635 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:38:05.882922 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:05.883273 (+ 351us) server_negotiation.cc:299] Negotiation successful
0924 11:38:05.883579 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 11:38:05.889744 23540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:05.866892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:40961 (local address 127.6.212.62:36825)
0924 11:38:05.867490 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:05.867494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:05.868880 (+ 1386us) server_negotiation.cc:407] Connection header received
0924 11:38:05.871473 (+ 2593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:05.871479 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:05.871570 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:05.871700 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:38:05.873808 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:05.874723 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:05.876790 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:05.877556 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:05.877691 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:05.882109 (+ 4418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:38:05.882115 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:38:05.882120 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:38:05.882160 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:38:05.886129 (+ 3969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:05.887112 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:05.887119 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:05.887124 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:05.887301 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:05.887917 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:05.887921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:05.887923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:05.888235 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:38:05.888470 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:05.889213 (+ 743us) server_negotiation.cc:299] Negotiation successful
0924 11:38:05.889451 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":284,"threads_started":1}
I20260924 11:38:05.890656 23541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:05.867670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.3:40961)
0924 11:38:05.868671 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:05.868718 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 11:38:05.871162 (+ 2444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:38:05.871981 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:38:05.871990 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:38:05.872620 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:38:05.873627 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:05.873648 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:05.874924 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:05.874928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:05.876598 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:05.876612 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:05.877824 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:05.877835 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:05.878183 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:05.878973 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:38:05.879004 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:38:05.881858 (+ 2854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:38:05.886276 (+ 4418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:38:05.886286 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:38:05.886305 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:38:05.886929 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:38:05.887563 (+ 634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:38:05.887567 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:38:05.887569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:38:05.887793 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:38:05.888461 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:38:05.888467 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:38:05.888849 (+ 382us) client_negotiation.cc:769] Sending connection context
0924 11:38:05.889091 (+ 242us) client_negotiation.cc:240] Negotiation successful
0924 11:38:05.889535 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"thread_start_us":806,"threads_started":1}
I20260924 11:38:05.892256 23537 heartbeater.cc:347] Connected to a master server at 127.6.212.62:36825
I20260924 11:38:05.892597 23537 heartbeater.cc:464] Registering TS with master...
I20260924 11:38:05.893404 23537 heartbeater.cc:511] Master 127.6.212.62:36825 requested a full tablet report, sending...
I20260924 11:38:05.894972 23089 ts_manager.cc:194] Registered new tserver with Master: 8832bc897e9d4f2191318fcdba73bb9d (127.6.212.3:46385)
I20260924 11:38:05.895896 23089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.3@KRBTEST.COM'} at 127.6.212.3:40961
I20260924 11:38:05.899659 6992 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:38:05.907191 23540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:05.900797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36772 (local address 127.6.212.62:36825)
0924 11:38:05.900997 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:05.901002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:05.901329 (+ 327us) server_negotiation.cc:407] Connection header received
0924 11:38:05.901542 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:05.901547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:05.901690 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:05.901823 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:38:05.902316 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:05.903049 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:05.904966 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:05.905451 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:05.905585 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:05.906185 (+ 600us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:38:05.906568 (+ 383us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 11:38:05.906941 (+ 373us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36772: token expired
Metrics: {"server-negotiator.queue_time_us":75}
W20260924 11:38:05.907325 23540 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36772: token expired
W20260924 11:38:05.908192 23011 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.6.212.62:36825)
I20260924 11:38:05.923476 23540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:05.908868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36782 (local address 127.6.212.62:36825)
0924 11:38:05.909131 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:05.909137 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:05.909158 (+ 21us) server_negotiation.cc:407] Connection header received
0924 11:38:05.909451 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:05.909464 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:05.909535 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:05.909652 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:38:05.911108 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:05.912058 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:05.913555 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:05.914002 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:05.914157 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:05.917304 (+ 3147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:38:05.917310 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:38:05.917313 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:38:05.917349 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:38:05.920595 (+ 3246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:05.921353 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:05.921361 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:05.921368 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:05.921484 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:05.921861 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:05.921867 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:05.921871 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:05.922209 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:38:05.922490 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:05.922870 (+ 380us) server_negotiation.cc:299] Negotiation successful
0924 11:38:05.923100 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 11:38:05.927058 23089 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36782:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:38:05.934387 23089 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:38:05.962507 23549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:05.948642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.2:36853 (local address 127.0.0.1:57426)
0924 11:38:05.953564 (+ 4922us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:05.953588 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 11:38:05.953765 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:38:05.954648 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:38:05.954655 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:38:05.954685 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:38:05.955192 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:05.955209 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:05.957249 (+ 2040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:05.957255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:05.959203 (+ 1948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:05.959226 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:05.961314 (+ 2088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:05.961321 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:05.961549 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:05.961610 (+ 61us) client_negotiation.cc:769] Sending connection context
0924 11:38:05.961775 (+ 165us) client_negotiation.cc:240] Negotiation successful
0924 11:38:05.961988 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4784,"mutex_wait_us":4,"thread_start_us":1790,"threads_started":1}
I20260924 11:38:05.962764 23550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:05.949245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57426 (local address 127.6.212.2:36853)
0924 11:38:05.950127 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:05.950142 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:05.953955 (+ 3813us) server_negotiation.cc:407] Connection header received
0924 11:38:05.954067 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:05.954075 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:05.954423 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:05.954627 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:38:05.955462 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:05.957098 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:05.959413 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:05.960771 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:05.960903 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:05.961148 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:05.962264 (+ 1116us) server_negotiation.cc:299] Negotiation successful
0924 11:38:05.962448 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":412,"threads_started":1}
I20260924 11:38:05.962821 23551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:05.952597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:46119 (local address 127.0.0.1:39866)
0924 11:38:05.953956 (+ 1359us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:05.953978 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 11:38:05.954097 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:38:05.954974 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:38:05.954978 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:38:05.955007 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:38:05.955428 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:05.955444 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:05.957364 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:05.957373 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:05.959796 (+ 2423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:05.959812 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:05.962018 (+ 2206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:05.962024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:05.962180 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:05.962224 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 11:38:05.962328 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 11:38:05.962458 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1221,"mutex_wait_us":24,"thread_start_us":1059,"threads_started":1}
I20260924 11:38:05.963129 23547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:05.948682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46106 (local address 127.6.212.3:46385)
0924 11:38:05.954671 (+ 5989us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:05.954679 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:05.954711 (+ 32us) server_negotiation.cc:407] Connection header received
0924 11:38:05.954819 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:05.954830 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:05.955073 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:05.955272 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:38:05.955994 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:05.957510 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:05.960046 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:05.961389 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:05.961544 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:05.961760 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:05.962549 (+ 789us) server_negotiation.cc:299] Negotiation successful
0924 11:38:05.962751 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5831,"thread_start_us":5657,"threads_started":1}
I20260924 11:38:05.963132 23548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:05.948598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.3:46385 (local address 127.0.0.1:46106)
0924 11:38:05.950910 (+ 2312us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:05.950967 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 11:38:05.951201 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:38:05.955259 (+ 4058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:38:05.955275 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:38:05.955302 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:38:05.955790 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:05.955804 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:05.957661 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:05.957668 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:05.959894 (+ 2226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:05.959907 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:05.961611 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:05.961617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:05.961807 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:05.961843 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 11:38:05.961951 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 11:38:05.962093 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1956,"thread_start_us":4642,"threads_started":1}
I20260924 11:38:05.964679 23552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:05.953402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39866 (local address 127.6.212.1:46119)
0924 11:38:05.954267 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:05.954277 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:05.954308 (+ 31us) server_negotiation.cc:407] Connection header received
0924 11:38:05.954432 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:05.954441 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:05.954697 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:05.954955 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:38:05.955655 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:05.957172 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:05.959970 (+ 2798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:05.961808 (+ 1838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:05.961938 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:05.962141 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:05.964145 (+ 2004us) server_negotiation.cc:299] Negotiation successful
0924 11:38:05.964339 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":530,"threads_started":1}
I20260924 11:38:05.966832 23341 tablet_service.cc:1467] Processing CreateTablet for tablet dfa84b724f4e49b782dd756fe1019bdc (DEFAULT_TABLE table=test-table [id=c1c3b8570a4c4ae1b25e9b7c10ededaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:38:05.966877 23472 tablet_service.cc:1467] Processing CreateTablet for tablet dfa84b724f4e49b782dd756fe1019bdc (DEFAULT_TABLE table=test-table [id=c1c3b8570a4c4ae1b25e9b7c10ededaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:38:05.968200 23341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfa84b724f4e49b782dd756fe1019bdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:38:05.968346 23472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfa84b724f4e49b782dd756fe1019bdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:38:05.967980 23209 tablet_service.cc:1467] Processing CreateTablet for tablet dfa84b724f4e49b782dd756fe1019bdc (DEFAULT_TABLE table=test-table [id=c1c3b8570a4c4ae1b25e9b7c10ededaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:38:05.969298 23209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfa84b724f4e49b782dd756fe1019bdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:38:05.983129 23553 tablet_bootstrap.cc:492] T dfa84b724f4e49b782dd756fe1019bdc P 63e7d703fab14977bd19cb32ff984f5e: Bootstrap starting.
I20260924 11:38:05.986963 23553 tablet_bootstrap.cc:654] T dfa84b724f4e49b782dd756fe1019bdc P 63e7d703fab14977bd19cb32ff984f5e: Neither blocks nor log segments found. Creating new log.
I20260924 11:38:05.988133 23555 tablet_bootstrap.cc:492] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d: Bootstrap starting.
I20260924 11:38:05.988444 23553 log.cc:824] T dfa84b724f4e49b782dd756fe1019bdc P 63e7d703fab14977bd19cb32ff984f5e: Log is configured to *not* fsync() on all Append() calls
I20260924 11:38:05.989914 23554 tablet_bootstrap.cc:492] T dfa84b724f4e49b782dd756fe1019bdc P 8832bc897e9d4f2191318fcdba73bb9d: Bootstrap starting.
I20260924 11:38:05.991298 23553 tablet_bootstrap.cc:492] T dfa84b724f4e49b782dd756fe1019bdc P 63e7d703fab14977bd19cb32ff984f5e: No bootstrap required, opened a new log
I20260924 11:38:05.991616 23553 ts_tablet_manager.cc:1397] T dfa84b724f4e49b782dd756fe1019bdc P 63e7d703fab14977bd19cb32ff984f5e: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260924 11:38:05.992911 23555 tablet_bootstrap.cc:654] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d: Neither blocks nor log segments found. Creating new log.
I20260924 11:38:05.993582 23554 tablet_bootstrap.cc:654] T dfa84b724f4e49b782dd756fe1019bdc P 8832bc897e9d4f2191318fcdba73bb9d: Neither blocks nor log segments found. Creating new log.
I20260924 11:38:05.994207 23555 log.cc:824] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d: Log is configured to *not* fsync() on all Append() calls
I20260924 11:38:05.994853 23554 log.cc:824] T dfa84b724f4e49b782dd756fe1019bdc P 8832bc897e9d4f2191318fcdba73bb9d: Log is configured to *not* fsync() on all Append() calls
I20260924 11:38:05.997864 23554 tablet_bootstrap.cc:492] T dfa84b724f4e49b782dd756fe1019bdc P 8832bc897e9d4f2191318fcdba73bb9d: No bootstrap required, opened a new log
I20260924 11:38:05.998158 23554 ts_tablet_manager.cc:1397] T dfa84b724f4e49b782dd756fe1019bdc P 8832bc897e9d4f2191318fcdba73bb9d: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.002s
I20260924 11:38:05.999940 23555 tablet_bootstrap.cc:492] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d: No bootstrap required, opened a new log
I20260924 11:38:06.000233 23555 ts_tablet_manager.cc:1397] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260924 11:38:06.000540 23553 raft_consensus.cc:348] T dfa84b724f4e49b782dd756fe1019bdc P 63e7d703fab14977bd19cb32ff984f5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8832bc897e9d4f2191318fcdba73bb9d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 46385 } } peers { permanent_uuid: "63e7d703fab14977bd19cb32ff984f5e" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36853 } } peers { permanent_uuid: "62c2c7c8728542e5a9f8f9851794a98d" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46119 } }
I20260924 11:38:06.000870 23553 raft_consensus.cc:374] T dfa84b724f4e49b782dd756fe1019bdc P 63e7d703fab14977bd19cb32ff984f5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:38:06.000932 23553 raft_consensus.cc:729] T dfa84b724f4e49b782dd756fe1019bdc P 63e7d703fab14977bd19cb32ff984f5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63e7d703fab14977bd19cb32ff984f5e, State: Initialized, Role: FOLLOWER
I20260924 11:38:06.001518 23553 consensus_queue.cc:260] T dfa84b724f4e49b782dd756fe1019bdc P 63e7d703fab14977bd19cb32ff984f5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8832bc897e9d4f2191318fcdba73bb9d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 46385 } } peers { permanent_uuid: "63e7d703fab14977bd19cb32ff984f5e" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36853 } } peers { permanent_uuid: "62c2c7c8728542e5a9f8f9851794a98d" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46119 } }
I20260924 11:38:06.002688 23406 heartbeater.cc:503] Master 127.6.212.62:36825 was elected leader, sending a full tablet report...
I20260924 11:38:06.003281 23553 ts_tablet_manager.cc:1428] T dfa84b724f4e49b782dd756fe1019bdc P 63e7d703fab14977bd19cb32ff984f5e: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
I20260924 11:38:06.008872 23554 raft_consensus.cc:348] T dfa84b724f4e49b782dd756fe1019bdc P 8832bc897e9d4f2191318fcdba73bb9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8832bc897e9d4f2191318fcdba73bb9d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 46385 } } peers { permanent_uuid: "63e7d703fab14977bd19cb32ff984f5e" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36853 } } peers { permanent_uuid: "62c2c7c8728542e5a9f8f9851794a98d" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46119 } }
I20260924 11:38:06.009431 23554 raft_consensus.cc:374] T dfa84b724f4e49b782dd756fe1019bdc P 8832bc897e9d4f2191318fcdba73bb9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:38:06.009548 23554 raft_consensus.cc:729] T dfa84b724f4e49b782dd756fe1019bdc P 8832bc897e9d4f2191318fcdba73bb9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8832bc897e9d4f2191318fcdba73bb9d, State: Initialized, Role: FOLLOWER
I20260924 11:38:06.010196 23554 consensus_queue.cc:260] T dfa84b724f4e49b782dd756fe1019bdc P 8832bc897e9d4f2191318fcdba73bb9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8832bc897e9d4f2191318fcdba73bb9d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 46385 } } peers { permanent_uuid: "63e7d703fab14977bd19cb32ff984f5e" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36853 } } peers { permanent_uuid: "62c2c7c8728542e5a9f8f9851794a98d" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46119 } }
I20260924 11:38:06.010128 23555 raft_consensus.cc:348] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8832bc897e9d4f2191318fcdba73bb9d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 46385 } } peers { permanent_uuid: "63e7d703fab14977bd19cb32ff984f5e" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36853 } } peers { permanent_uuid: "62c2c7c8728542e5a9f8f9851794a98d" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46119 } }
I20260924 11:38:06.010488 23555 raft_consensus.cc:374] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:38:06.010589 23555 raft_consensus.cc:729] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62c2c7c8728542e5a9f8f9851794a98d, State: Initialized, Role: FOLLOWER
I20260924 11:38:06.011394 23537 heartbeater.cc:503] Master 127.6.212.62:36825 was elected leader, sending a full tablet report...
I20260924 11:38:06.011234 23555 consensus_queue.cc:260] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8832bc897e9d4f2191318fcdba73bb9d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 46385 } } peers { permanent_uuid: "63e7d703fab14977bd19cb32ff984f5e" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36853 } } peers { permanent_uuid: "62c2c7c8728542e5a9f8f9851794a98d" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46119 } }
I20260924 11:38:06.012257 23554 ts_tablet_manager.cc:1428] T dfa84b724f4e49b782dd756fe1019bdc P 8832bc897e9d4f2191318fcdba73bb9d: Time spent starting tablet: real 0.014s user 0.012s sys 0.002s
I20260924 11:38:06.013187 23555 ts_tablet_manager.cc:1428] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
W20260924 11:38:06.087248 23275 tablet.cc:2369] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:38:06.111426 23407 tablet.cc:2369] T dfa84b724f4e49b782dd756fe1019bdc P 63e7d703fab14977bd19cb32ff984f5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:38:06.115485 23538 tablet.cc:2369] T dfa84b724f4e49b782dd756fe1019bdc P 8832bc897e9d4f2191318fcdba73bb9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:38:06.130896 23561 raft_consensus.cc:482] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:38:06.131250 23561 raft_consensus.cc:504] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8832bc897e9d4f2191318fcdba73bb9d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 46385 } } peers { permanent_uuid: "63e7d703fab14977bd19cb32ff984f5e" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36853 } } peers { permanent_uuid: "62c2c7c8728542e5a9f8f9851794a98d" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46119 } }
I20260924 11:38:06.133198 23561 leader_election.cc:288] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8832bc897e9d4f2191318fcdba73bb9d (127.6.212.3:46385), 63e7d703fab14977bd19cb32ff984f5e (127.6.212.2:36853)
I20260924 11:38:06.140497 23550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:06.133850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:43691 (local address 127.6.212.2:36853)
0924 11:38:06.134087 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:06.134096 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:06.134761 (+ 665us) server_negotiation.cc:407] Connection header received
0924 11:38:06.134961 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:06.134967 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:06.135026 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:06.135127 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:38:06.135821 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:06.136570 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:06.138599 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:06.139446 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:06.139593 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:06.139715 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:06.140074 (+ 359us) server_negotiation.cc:299] Negotiation successful
0924 11:38:06.140211 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 11:38:06.140493 23547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:06.133640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:37869 (local address 127.6.212.3:46385)
0924 11:38:06.133877 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:06.133886 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:06.134677 (+ 791us) server_negotiation.cc:407] Connection header received
0924 11:38:06.134971 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:06.134977 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:06.135044 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:06.135142 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:38:06.135828 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:06.136568 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:06.138587 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:06.139450 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:06.139598 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:06.139711 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:06.140062 (+ 351us) server_negotiation.cc:299] Negotiation successful
0924 11:38:06.140198 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 11:38:06.140657 23562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:06.133683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.3:46385 (local address 127.6.212.1:37869)
0924 11:38:06.134551 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:06.134569 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:38:06.134748 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:38:06.135156 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:38:06.135160 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:38:06.135182 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:38:06.135685 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:06.135694 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:06.136702 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:06.136710 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:06.138433 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:06.138445 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:06.139681 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:06.139687 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:06.139860 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:06.139905 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 11:38:06.140036 (+ 131us) client_negotiation.cc:240] Negotiation successful
0924 11:38:06.140198 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"mutex_wait_us":27,"thread_start_us":470,"threads_started":1}
I20260924 11:38:06.140960 23563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:06.133796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.2:36853 (local address 127.6.212.1:43691)
0924 11:38:06.134669 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:06.134683 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 11:38:06.134794 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:38:06.135142 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:38:06.135146 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:38:06.135169 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:38:06.135632 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:06.135648 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:06.136711 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:06.136717 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:06.138439 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:06.138451 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:06.139716 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:06.139723 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:06.139871 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:06.139920 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 11:38:06.140048 (+ 128us) client_negotiation.cc:240] Negotiation successful
0924 11:38:06.140210 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"mutex_wait_us":59,"thread_start_us":496,"threads_started":1}
I20260924 11:38:06.141930 23492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa84b724f4e49b782dd756fe1019bdc" candidate_uuid: "62c2c7c8728542e5a9f8f9851794a98d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8832bc897e9d4f2191318fcdba73bb9d" is_pre_election: true
I20260924 11:38:06.142053 23361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa84b724f4e49b782dd756fe1019bdc" candidate_uuid: "62c2c7c8728542e5a9f8f9851794a98d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63e7d703fab14977bd19cb32ff984f5e" is_pre_election: true
I20260924 11:38:06.142549 23492 raft_consensus.cc:2454] T dfa84b724f4e49b782dd756fe1019bdc P 8832bc897e9d4f2191318fcdba73bb9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62c2c7c8728542e5a9f8f9851794a98d in term 0.
I20260924 11:38:06.142629 23361 raft_consensus.cc:2454] T dfa84b724f4e49b782dd756fe1019bdc P 63e7d703fab14977bd19cb32ff984f5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62c2c7c8728542e5a9f8f9851794a98d in term 0.
I20260924 11:38:06.143306 23165 leader_election.cc:302] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62c2c7c8728542e5a9f8f9851794a98d, 63e7d703fab14977bd19cb32ff984f5e; no voters:
I20260924 11:38:06.143805 23561 raft_consensus.cc:2781] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:38:06.143963 23561 raft_consensus.cc:482] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:38:06.144069 23561 raft_consensus.cc:3037] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:38:06.146284 23561 raft_consensus.cc:504] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8832bc897e9d4f2191318fcdba73bb9d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 46385 } } peers { permanent_uuid: "63e7d703fab14977bd19cb32ff984f5e" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36853 } } peers { permanent_uuid: "62c2c7c8728542e5a9f8f9851794a98d" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46119 } }
I20260924 11:38:06.146943 23561 leader_election.cc:288] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [CANDIDATE]: Term 1 election: Requested vote from peers 8832bc897e9d4f2191318fcdba73bb9d (127.6.212.3:46385), 63e7d703fab14977bd19cb32ff984f5e (127.6.212.2:36853)
I20260924 11:38:06.147641 23492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa84b724f4e49b782dd756fe1019bdc" candidate_uuid: "62c2c7c8728542e5a9f8f9851794a98d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8832bc897e9d4f2191318fcdba73bb9d"
I20260924 11:38:06.147641 23361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa84b724f4e49b782dd756fe1019bdc" candidate_uuid: "62c2c7c8728542e5a9f8f9851794a98d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63e7d703fab14977bd19cb32ff984f5e"
I20260924 11:38:06.147879 23361 raft_consensus.cc:3037] T dfa84b724f4e49b782dd756fe1019bdc P 63e7d703fab14977bd19cb32ff984f5e [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:38:06.147895 23492 raft_consensus.cc:3037] T dfa84b724f4e49b782dd756fe1019bdc P 8832bc897e9d4f2191318fcdba73bb9d [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:38:06.151299 23492 raft_consensus.cc:2454] T dfa84b724f4e49b782dd756fe1019bdc P 8832bc897e9d4f2191318fcdba73bb9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62c2c7c8728542e5a9f8f9851794a98d in term 1.
I20260924 11:38:06.151299 23361 raft_consensus.cc:2454] T dfa84b724f4e49b782dd756fe1019bdc P 63e7d703fab14977bd19cb32ff984f5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62c2c7c8728542e5a9f8f9851794a98d in term 1.
I20260924 11:38:06.151871 23164 leader_election.cc:302] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62c2c7c8728542e5a9f8f9851794a98d, 8832bc897e9d4f2191318fcdba73bb9d; no voters:
I20260924 11:38:06.152247 23561 raft_consensus.cc:2781] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:38:06.152916 23561 raft_consensus.cc:686] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [term 1 LEADER]: Becoming Leader. State: Replica: 62c2c7c8728542e5a9f8f9851794a98d, State: Running, Role: LEADER
I20260924 11:38:06.153893 23561 consensus_queue.cc:237] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8832bc897e9d4f2191318fcdba73bb9d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 46385 } } peers { permanent_uuid: "63e7d703fab14977bd19cb32ff984f5e" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36853 } } peers { permanent_uuid: "62c2c7c8728542e5a9f8f9851794a98d" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46119 } }
I20260924 11:38:06.158959 23088 catalog_manager.cc:5495] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d reported cstate change: term changed from 0 to 1, leader changed from <none> to 62c2c7c8728542e5a9f8f9851794a98d (127.6.212.1). New cstate: current_term: 1 leader_uuid: "62c2c7c8728542e5a9f8f9851794a98d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8832bc897e9d4f2191318fcdba73bb9d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 46385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63e7d703fab14977bd19cb32ff984f5e" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "62c2c7c8728542e5a9f8f9851794a98d" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46119 } health_report { overall_health: HEALTHY } } }
I20260924 11:38:06.201470 23552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:06.195733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39880 (local address 127.6.212.1:46119)
0924 11:38:06.195964 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:06.195969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:06.195991 (+ 22us) server_negotiation.cc:407] Connection header received
0924 11:38:06.196245 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:06.196249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:06.196347 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:06.196508 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:38:06.196889 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:06.197731 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:06.199200 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:06.199690 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:06.199830 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:06.200382 (+ 552us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:38:06.200884 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:38:06.201054 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:06.201108 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 11:38:06.201187 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 11:38:06.210148 23492 raft_consensus.cc:1261] T dfa84b724f4e49b782dd756fe1019bdc P 8832bc897e9d4f2191318fcdba73bb9d [term 1 FOLLOWER]: Refusing update from remote peer 62c2c7c8728542e5a9f8f9851794a98d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:38:06.210145 23361 raft_consensus.cc:1261] T dfa84b724f4e49b782dd756fe1019bdc P 63e7d703fab14977bd19cb32ff984f5e [term 1 FOLLOWER]: Refusing update from remote peer 62c2c7c8728542e5a9f8f9851794a98d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:38:06.211149 23561 consensus_queue.cc:1035] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8832bc897e9d4f2191318fcdba73bb9d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 46385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:38:06.211478 23564 consensus_queue.cc:1035] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63e7d703fab14977bd19cb32ff984f5e" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:38:06.221203 23566 mvcc.cc:205] Tried to move back new op lower bound from 7332863533905809408 to 7332863533690986496. Current Snapshot: MvccSnapshot[applied={T|T < 7332863533905809408}]
I20260924 11:38:06.227569 23568 mvcc.cc:205] Tried to move back new op lower bound from 7332863533905809408 to 7332863533690986496. Current Snapshot: MvccSnapshot[applied={T|T < 7332863533905809408 or (T in {7332863533905809408})}]
I20260924 11:38:06.243153 23088 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:38:06.243518 23088 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:38:06.247541 23088 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf: Sending DeleteTablet for 3 replicas of tablet dfa84b724f4e49b782dd756fe1019bdc
I20260924 11:38:06.249228 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 23126
I20260924 11:38:06.249179 23341 tablet_service.cc:1514] Processing DeleteTablet for tablet dfa84b724f4e49b782dd756fe1019bdc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:38:06 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:57426
I20260924 11:38:06.249385 23472 tablet_service.cc:1514] Processing DeleteTablet for tablet dfa84b724f4e49b782dd756fe1019bdc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:38:06 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:46106
I20260924 11:38:06.250435 23577 tablet_replica.cc:330] T dfa84b724f4e49b782dd756fe1019bdc P 8832bc897e9d4f2191318fcdba73bb9d: stopping tablet replica
I20260924 11:38:06.251322 23577 raft_consensus.cc:2229] T dfa84b724f4e49b782dd756fe1019bdc P 8832bc897e9d4f2191318fcdba73bb9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:38:06.251547 23576 tablet_replica.cc:330] T dfa84b724f4e49b782dd756fe1019bdc P 63e7d703fab14977bd19cb32ff984f5e: stopping tablet replica
I20260924 11:38:06.251552 23209 tablet_service.cc:1514] Processing DeleteTablet for tablet dfa84b724f4e49b782dd756fe1019bdc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:38:06 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:39866
I20260924 11:38:06.251953 23577 raft_consensus.cc:2258] T dfa84b724f4e49b782dd756fe1019bdc P 8832bc897e9d4f2191318fcdba73bb9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:38:06.252081 23576 raft_consensus.cc:2229] T dfa84b724f4e49b782dd756fe1019bdc P 63e7d703fab14977bd19cb32ff984f5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:38:06.252625 23576 raft_consensus.cc:2258] T dfa84b724f4e49b782dd756fe1019bdc P 63e7d703fab14977bd19cb32ff984f5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:38:06.254958 23577 ts_tablet_manager.cc:1905] T dfa84b724f4e49b782dd756fe1019bdc P 8832bc897e9d4f2191318fcdba73bb9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:38:06.263077 23576 ts_tablet_manager.cc:1905] T dfa84b724f4e49b782dd756fe1019bdc P 63e7d703fab14977bd19cb32ff984f5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:38:06.263892 23577 ts_tablet_manager.cc:1918] T dfa84b724f4e49b782dd756fe1019bdc P 8832bc897e9d4f2191318fcdba73bb9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:38:06.264079 23577 log.cc:1196] T dfa84b724f4e49b782dd756fe1019bdc P 8832bc897e9d4f2191318fcdba73bb9d: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-2/wal/wals/dfa84b724f4e49b782dd756fe1019bdc
I20260924 11:38:06.264659 23577 ts_tablet_manager.cc:1939] T dfa84b724f4e49b782dd756fe1019bdc P 8832bc897e9d4f2191318fcdba73bb9d: Deleting consensus metadata
I20260924 11:38:06.266858 23075 catalog_manager.cc:4841] TS 8832bc897e9d4f2191318fcdba73bb9d (127.6.212.3:46385): tablet dfa84b724f4e49b782dd756fe1019bdc (table test-table [id=c1c3b8570a4c4ae1b25e9b7c10ededaa]) successfully deleted
I20260924 11:38:06.270330 23578 tablet_replica.cc:330] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d: stopping tablet replica
I20260924 11:38:06.270820 23578 raft_consensus.cc:2229] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:38:06.271730 23578 raft_consensus.cc:2258] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:38:06.271977 23576 ts_tablet_manager.cc:1918] T dfa84b724f4e49b782dd756fe1019bdc P 63e7d703fab14977bd19cb32ff984f5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:38:06.272200 23576 log.cc:1196] T dfa84b724f4e49b782dd756fe1019bdc P 63e7d703fab14977bd19cb32ff984f5e: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-1/wal/wals/dfa84b724f4e49b782dd756fe1019bdc
I20260924 11:38:06.272449 23552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:06.250798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39890 (local address 127.6.212.1:46119)
0924 11:38:06.250991 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:06.250999 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:06.251021 (+ 22us) server_negotiation.cc:407] Connection header received
0924 11:38:06.251099 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:06.251105 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:06.251200 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:06.251307 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:38:06.253377 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:06.254678 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:06.256836 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:06.257584 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:06.257710 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:06.261669 (+ 3959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:38:06.261686 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:38:06.261704 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:38:06.261769 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:38:06.266767 (+ 4998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:06.267907 (+ 1140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:06.267918 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:06.267921 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:06.268129 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:06.268805 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:06.268812 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:06.268816 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:06.269744 (+ 928us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:38:06.270094 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:06.271323 (+ 1229us) server_negotiation.cc:299] Negotiation successful
0924 11:38:06.271658 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 11:38:06.273031 23576 ts_tablet_manager.cc:1939] T dfa84b724f4e49b782dd756fe1019bdc P 63e7d703fab14977bd19cb32ff984f5e: Deleting consensus metadata
I20260924 11:38:06.273237 23269 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:38:06.275101 23076 catalog_manager.cc:4841] TS 63e7d703fab14977bd19cb32ff984f5e (127.6.212.2:36853): tablet dfa84b724f4e49b782dd756fe1019bdc (table test-table [id=c1c3b8570a4c4ae1b25e9b7c10ededaa]) successfully deleted
I20260924 11:38:06.274173 23578 ts_tablet_manager.cc:1905] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:38:06.373090 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 23126
W20260924 11:38:06.393424 23076 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.212.1:46119 (error 108)
I20260924 11:38:06.393514 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 23278
I20260924 11:38:06.395241 23549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:06.394830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:46119 (local address 127.0.0.1:40172)
0924 11:38:06.395000 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:06.395105 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.1:46119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260924 11:38:06.395557 23076 catalog_manager.cc:4584] TS 62c2c7c8728542e5a9f8f9851794a98d (127.6.212.1:46119): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dfa84b724f4e49b782dd756fe1019bdc: Network error: Client connection negotiation failed: client connection to 127.6.212.1:46119: connect: Connection refused (error 111)
I20260924 11:38:06.408547 23550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:06.394416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50980 (local address 127.6.212.2:36853)
0924 11:38:06.394602 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:06.394608 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:06.394723 (+ 115us) server_negotiation.cc:407] Connection header received
0924 11:38:06.394953 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:06.394957 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:06.395037 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:06.395164 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:38:06.396707 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:06.397476 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:06.398787 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:06.399223 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:06.399361 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:06.402048 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:38:06.402063 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:38:06.402078 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:38:06.402111 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:38:06.405041 (+ 2930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:06.405967 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:06.405976 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:06.405979 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:06.406114 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:06.406541 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:06.406548 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:06.406552 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:06.406981 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:38:06.407340 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:06.407863 (+ 523us) server_negotiation.cc:299] Negotiation successful
0924 11:38:06.408111 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 11:38:06.409428 23401 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:38:06.449160 23549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:06.448589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:46119 (local address 127.0.0.1:40188)
0924 11:38:06.448897 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:06.448997 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.1:46119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169}
I20260924 11:38:06.450273 23549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:06.449897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:46119 (local address 127.0.0.1:40198)
0924 11:38:06.450047 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:06.450147 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.1:46119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 11:38:06.502225 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 23278
I20260924 11:38:06.511073 23549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:06.510510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:46119 (local address 127.0.0.1:40214)
0924 11:38:06.510769 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:06.510881 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.1:46119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260924 11:38:06.512578 23549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:06.512007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:46119 (local address 127.0.0.1:40222)
0924 11:38:06.512199 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:06.512399 (+ 200us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.1:46119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 11:38:06.520686 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 23409
I20260924 11:38:06.534832 23547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:06.521595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48688 (local address 127.6.212.3:46385)
0924 11:38:06.521757 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:06.521762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:06.521779 (+ 17us) server_negotiation.cc:407] Connection header received
0924 11:38:06.521953 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:06.521958 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:06.522025 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:06.522109 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:38:06.523634 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:06.524346 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:06.525613 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:06.526021 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:06.526115 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:06.529145 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:38:06.529161 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:38:06.529177 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:38:06.529219 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:38:06.532149 (+ 2930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:06.532929 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:06.532936 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:06.532938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:06.533086 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:06.533507 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:06.533510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:06.533512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:06.533873 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:38:06.534093 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:06.534367 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 11:38:06.534539 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 11:38:06.535526 23532 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:38:06.593097 23549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:06.592451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:46119 (local address 127.0.0.1:40232)
0924 11:38:06.592738 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:06.592877 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.1:46119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260924 11:38:06.594540 23549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:06.593902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:46119 (local address 127.0.0.1:40246)
0924 11:38:06.594137 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:06.594394 (+ 257us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.1:46119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260924 11:38:06.628443 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 23409
I20260924 11:38:06.645756 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 23056
I20260924 11:38:06.646907 23119 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:38:06.766026 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 23056
I20260924 11:38:26.790195 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:36825
--webserver_interface=127.6.212.62
--webserver_port=36195
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:36245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:36825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:38:27.157951 23584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:38:27.158303 23584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:38:27.158465 23584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:38:27.166380 23584 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 11:38:27.166507 23584 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 11:38:27.166548 23584 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 11:38:27.166698 23584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:38:27.166828 23584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:38:27.166944 23584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:38:27.167030 23584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:38:27.167119 23584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:38:27.178064 23584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:36245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:36825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:36825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.212.62
--webserver_port=36195
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:38:27.179644 23584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:38:27.181202 23584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:38:27.194890 23590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:38:27.195005 23589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:38:27.194890 23592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:38:27.195423 23584 server_base.cc:1034] running on GCE node
I20260924 11:38:27.196858 23584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:38:27.198623 23584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:38:27.199934 23584 hybrid_clock.cc:648] HybridClock initialized: now 1790249907199843 us; error 69 us; skew 500 ppm
I20260924 11:38:27.200330 23584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:38:27 dist-test-slave-cq7c krb5kdc[22504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249907, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:38:27.210290 23584 init.cc:377] Logged in from keytab as kudu/127.6.212.62@KRBTEST.COM (short username kudu)
I20260924 11:38:27.212260 23584 webserver.cc:460] Webserver started at http://127.6.212.62:36195/ using document root <none> and password file <none>
I20260924 11:38:27.213086 23584 fs_manager.cc:359] Metadata directory not provided
I20260924 11:38:27.213240 23584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:38:27.219619 23584 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 11:38:27.223242 23599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:27.225164 23584 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 11:38:27.225419 23584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "ac21188da6be46928a99557e485926bf"
format_stamp: "Formatted at 2026-09-24 11:37:41 on dist-test-slave-cq7c"
server_key: "c9a674c6ddf3b9c9bfe32ff0b221b7c1"
server_key_iv: "5aeadd200445eafd60a4ecce6a1dfaa2"
server_key_version: "encryptionkey@0"
I20260924 11:38:27.226522 23584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:38:27.252245 23584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:38:27.257440 23584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:38:27.257782 23584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:38:27.279944 23584 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.62:36825
I20260924 11:38:27.279978 23650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.62:36825 every 8 connection(s)
I20260924 11:38:27.281723 23584 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:38:27.289552 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 23584
I20260924 11:38:27.290320 23651 sys_catalog.cc:263] Verifying existing consensus state
I20260924 11:38:27.290800 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.1:46119
--local_ip_for_outbound_sockets=127.6.212.1
--tserver_master_addrs=127.6.212.62:36825
--webserver_port=41729
--webserver_interface=127.6.212.1
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab
--principal=kudu/127.6.212.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:36245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:38:27.296805 23651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf: Bootstrap starting.
I20260924 11:38:27.322574 23651 log.cc:824] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf: Log is configured to *not* fsync() on all Append() calls
I20260924 11:38:27.342263 23651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 11:38:27.342967 23651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf: Bootstrap complete.
I20260924 11:38:27.353742 23651 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac21188da6be46928a99557e485926bf" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36825 } }
I20260924 11:38:27.354831 23651 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac21188da6be46928a99557e485926bf, State: Initialized, Role: FOLLOWER
I20260924 11:38:27.355737 23651 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac21188da6be46928a99557e485926bf" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36825 } }
I20260924 11:38:27.356030 23651 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:38:27.356169 23651 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:38:27.356372 23651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [term 2 FOLLOWER]: Advancing to term 3
I20260924 11:38:27.361074 23651 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac21188da6be46928a99557e485926bf" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36825 } }
I20260924 11:38:27.361724 23651 leader_election.cc:302] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac21188da6be46928a99557e485926bf; no voters:
I20260924 11:38:27.362522 23651 leader_election.cc:288] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 11:38:27.362690 23655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [term 3 FOLLOWER]: Leader election won for term 3
I20260924 11:38:27.364120 23655 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [term 3 LEADER]: Becoming Leader. State: Replica: ac21188da6be46928a99557e485926bf, State: Running, Role: LEADER
I20260924 11:38:27.364825 23655 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac21188da6be46928a99557e485926bf" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36825 } }
I20260924 11:38:27.365309 23651 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:38:27.367853 23656 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ac21188da6be46928a99557e485926bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac21188da6be46928a99557e485926bf" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36825 } } }
I20260924 11:38:27.368383 23656 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [sys.catalog]: This master's current role is: LEADER
I20260924 11:38:27.368716 23657 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac21188da6be46928a99557e485926bf. Latest consensus state: current_term: 3 leader_uuid: "ac21188da6be46928a99557e485926bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac21188da6be46928a99557e485926bf" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36825 } } }
I20260924 11:38:27.369191 23657 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf [sys.catalog]: This master's current role is: LEADER
I20260924 11:38:27.371783 23660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:38:27.381994 23660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:38:27.383719 23660 catalog_manager.cc:1259] Loaded cluster ID: 0b046aa47f3c42a38c8a9b1a5dcd2232
I20260924 11:38:27.383920 23660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:38:27.389597 23660 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:38:27.391613 23660 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf: Loaded TSK: 0
I20260924 11:38:27.391881 23660 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ac21188da6be46928a99557e485926bf: Loaded TSK: 1
F20260924 11:38:27.393067 23660 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fcba5f799ad google::LogMessage::Fail() at ??:0
@ 0x7fcba5f7c88a google::LogMessage::SendToLog() at ??:0
@ 0x7fcba5f794de google::LogMessage::Flush() at ??:0
@ 0x7fcba5f7aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fcba9cdd7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fcba9ccc73e kudu::security::TokenSigner::ImportKeys() at ??:0
I20260924 11:38:27.407558 23673 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7fcbbb9a81c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fcbbb9a7783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fcbbba332a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fcbbba330c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fcbbba7d5e9 std::function<>::operator()() at ??:0
@ 0x7fcbbb9abcdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fcbbb9a9c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fcbbba31610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fcbbba31439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fcba8bc5345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fcba8bc891c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fcba8bc8799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fcba8ba3eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fcbabe7d6db start_thread at ??:0
@ 0x7fcba4e1361f clone at ??:0
W20260924 11:38:27.741698 23653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:38:27.742018 23653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:38:27.742206 23653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:38:27.750088 23653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:38:27.750242 23653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:38:27.750445 23653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.1
I20260924 11:38:27.761436 23653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:36245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab
--principal=kudu/127.6.212.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.1:46119
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.212.1
--webserver_port=41729
--tserver_master_addrs=127.6.212.62:36825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.1
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:38:27.762917 23653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:38:27.764469 23653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:38:27.780000 23699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:38:27.780071 23698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:38:27.780046 23701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:38:27.781255 23653 server_base.cc:1034] running on GCE node
I20260924 11:38:27.781999 23653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:38:27.783215 23653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:38:27.784488 23653 hybrid_clock.cc:648] HybridClock initialized: now 1790249907784451 us; error 67 us; skew 500 ppm
I20260924 11:38:27.784884 23653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:38:27 dist-test-slave-cq7c krb5kdc[22504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249907, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:38:27.788563 23653 init.cc:377] Logged in from keytab as kudu/127.6.212.1@KRBTEST.COM (short username kudu)
I20260924 11:38:27.790279 23653 webserver.cc:460] Webserver started at http://127.6.212.1:41729/ using document root <none> and password file <none>
I20260924 11:38:27.791059 23653 fs_manager.cc:359] Metadata directory not provided
I20260924 11:38:27.791218 23653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:38:27.797498 23653 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 11:38:27.801229 23708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:27.803449 23653 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260924 11:38:27.803681 23653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-0/wal
uuid: "62c2c7c8728542e5a9f8f9851794a98d"
format_stamp: "Formatted at 2026-09-24 11:37:41 on dist-test-slave-cq7c"
server_key: "27ba1496484442fc22dfffff75b3fcc8"
server_key_iv: "95acc6f22192c558b523389e9c3e5bab"
server_key_version: "encryptionkey@0"
I20260924 11:38:27.804764 23653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:38:27.821400 23653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:38:27.826162 23653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:38:27.826419 23653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:38:27.827651 23653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:38:27.829788 23715 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260924 11:38:27.837028 23653 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 11:38:27.837231 23653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.001s sys 0.000s
I20260924 11:38:27.837407 23653 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 11:38:27.841136 23653 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 11:38:27.841282 23653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.002s sys 0.000s
I20260924 11:38:27.841696 23715 tablet_bootstrap.cc:492] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d: Bootstrap starting.
I20260924 11:38:27.852092 23715 log.cc:824] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d: Log is configured to *not* fsync() on all Append() calls
I20260924 11:38:27.871142 23715 tablet_bootstrap.cc:492] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 11:38:27.872144 23715 tablet_bootstrap.cc:492] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d: Bootstrap complete.
I20260924 11:38:27.872776 23715 ts_tablet_manager.cc:1397] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d: Time spent bootstrapping tablet: real 0.031s user 0.022s sys 0.007s
I20260924 11:38:27.883718 23715 raft_consensus.cc:348] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8832bc897e9d4f2191318fcdba73bb9d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 46385 } } peers { permanent_uuid: "63e7d703fab14977bd19cb32ff984f5e" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36853 } } peers { permanent_uuid: "62c2c7c8728542e5a9f8f9851794a98d" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46119 } }
I20260924 11:38:27.884923 23715 raft_consensus.cc:729] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62c2c7c8728542e5a9f8f9851794a98d, State: Initialized, Role: FOLLOWER
I20260924 11:38:27.885833 23715 consensus_queue.cc:260] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8832bc897e9d4f2191318fcdba73bb9d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 46385 } } peers { permanent_uuid: "63e7d703fab14977bd19cb32ff984f5e" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36853 } } peers { permanent_uuid: "62c2c7c8728542e5a9f8f9851794a98d" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46119 } }
I20260924 11:38:27.887965 23715 ts_tablet_manager.cc:1428] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d: Time spent starting tablet: real 0.015s user 0.011s sys 0.005s
I20260924 11:38:27.890827 23653 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.1:46119
I20260924 11:38:27.890908 23823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.1:46119 every 8 connection(s)
I20260924 11:38:27.892247 23653 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
I20260924 11:38:27.894327 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 23653
I20260924 11:38:27.895117 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.2:36853
--local_ip_for_outbound_sockets=127.6.212.2
--tserver_master_addrs=127.6.212.62:36825
--webserver_port=41293
--webserver_interface=127.6.212.2
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab
--principal=kudu/127.6.212.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:36245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:38:27.897768 23826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:27.895515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.1:39671)
0924 11:38:27.896516 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:27.896811 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":446,"threads_started":1}
W20260924 11:38:27.900380 23824 heartbeater.cc:644] Failed to heartbeat to 127.6.212.62:36825 (0 consecutive failures): Network error: Failed to ping master at 127.6.212.62:36825: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
I20260924 11:38:27.902489 23826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:27.900993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.1:45493)
0924 11:38:27.902294 (+ 1301us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:27.902372 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1193}
I20260924 11:38:27.903714 23826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:27.903022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.1:39553)
0924 11:38:27.903187 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:27.903569 (+ 382us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 11:38:27.904044 23824 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 11:38:28.251777 23827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:38:28.252103 23827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:38:28.252219 23827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:38:28.260208 23827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:38:28.260353 23827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:38:28.260520 23827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.2
I20260924 11:38:28.271785 23827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:36245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab
--principal=kudu/127.6.212.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.2:36853
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.212.2
--webserver_port=41293
--tserver_master_addrs=127.6.212.62:36825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.2
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:38:28.273161 23827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:38:28.274578 23827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:38:28.289319 23832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:38:28.289265 23835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:38:28.289354 23833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:38:28.290772 23827 server_base.cc:1034] running on GCE node
I20260924 11:38:28.291676 23827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:38:28.292882 23827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:38:28.294166 23827 hybrid_clock.cc:648] HybridClock initialized: now 1790249908294126 us; error 33 us; skew 500 ppm
I20260924 11:38:28.294544 23827 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:38:28 dist-test-slave-cq7c krb5kdc[22504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249908, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:38:28.298074 23827 init.cc:377] Logged in from keytab as kudu/127.6.212.2@KRBTEST.COM (short username kudu)
I20260924 11:38:28.299858 23827 webserver.cc:460] Webserver started at http://127.6.212.2:41293/ using document root <none> and password file <none>
I20260924 11:38:28.300813 23827 fs_manager.cc:359] Metadata directory not provided
I20260924 11:38:28.300972 23827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:38:28.307327 23827 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 11:38:28.310978 23842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:28.312929 23827 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 11:38:28.313153 23827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-1/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-1/wal
uuid: "63e7d703fab14977bd19cb32ff984f5e"
format_stamp: "Formatted at 2026-09-24 11:37:42 on dist-test-slave-cq7c"
server_key: "e97b9476f1007d167819ee0e85b6c5a6"
server_key_iv: "ffb18584d4aa4d09a8742220718447aa"
server_key_version: "encryptionkey@0"
I20260924 11:38:28.314277 23827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:38:28.344704 23827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:38:28.349613 23827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:38:28.349931 23827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:38:28.351059 23827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:38:28.352769 23827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:38:28.352897 23827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:28.353034 23827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:38:28.353134 23827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:28.390573 23827 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.2:36853
I20260924 11:38:28.390627 23954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.2:36853 every 8 connection(s)
I20260924 11:38:28.392138 23827 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
I20260924 11:38:28.394443 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 23827
I20260924 11:38:28.395243 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.3:46385
--local_ip_for_outbound_sockets=127.6.212.3
--tserver_master_addrs=127.6.212.62:36825
--webserver_port=32777
--webserver_interface=127.6.212.3
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab
--principal=kudu/127.6.212.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:36245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:38:28.398598 23957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:28.396229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.2:45441)
0924 11:38:28.397311 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:28.397571 (+ 260us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":403,"threads_started":1}
W20260924 11:38:28.399554 23955 heartbeater.cc:644] Failed to heartbeat to 127.6.212.62:36825 (0 consecutive failures): Network error: Failed to ping master at 127.6.212.62:36825: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
I20260924 11:38:28.400705 23957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:28.400276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.2:56667)
0924 11:38:28.400479 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:28.400570 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260924 11:38:28.401801 23957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:28.401440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.2:47983)
0924 11:38:28.401589 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:28.401674 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 11:38:28.402139 23955 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 11:38:28.754925 23958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:38:28.755249 23958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:38:28.755347 23958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:38:28.763126 23958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:38:28.763268 23958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:38:28.763461 23958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.3
I20260924 11:38:28.774180 23958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:36245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab
--principal=kudu/127.6.212.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.3:46385
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.212.3
--webserver_port=32777
--tserver_master_addrs=127.6.212.62:36825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.3
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:38:28.775830 23958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:38:28.777339 23958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:38:28.792111 23963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:38:28.792184 23966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:38:28.792135 23964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:38:28.793382 23958 server_base.cc:1034] running on GCE node
I20260924 11:38:28.794361 23958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:38:28.795691 23958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:38:28.796941 23958 hybrid_clock.cc:648] HybridClock initialized: now 1790249908796861 us; error 71 us; skew 500 ppm
I20260924 11:38:28.797394 23958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:38:28 dist-test-slave-cq7c krb5kdc[22504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249908, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:38:28.801399 23958 init.cc:377] Logged in from keytab as kudu/127.6.212.3@KRBTEST.COM (short username kudu)
I20260924 11:38:28.803439 23958 webserver.cc:460] Webserver started at http://127.6.212.3:32777/ using document root <none> and password file <none>
I20260924 11:38:28.804255 23958 fs_manager.cc:359] Metadata directory not provided
I20260924 11:38:28.804425 23958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:38:28.811589 23958 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 11:38:28.815053 23973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:28.816964 23958 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 11:38:28.817283 23958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-2/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-2/wal
uuid: "8832bc897e9d4f2191318fcdba73bb9d"
format_stamp: "Formatted at 2026-09-24 11:37:42 on dist-test-slave-cq7c"
server_key: "d171b7ddb0334a886d3649ce236fb813"
server_key_iv: "115f046321080e9ced9fc85257c19380"
server_key_version: "encryptionkey@0"
I20260924 11:38:28.818499 23958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:38:28.838464 23958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:38:28.843705 23958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:38:28.844023 23958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:38:28.845263 23958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:38:28.846748 23958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:38:28.846875 23958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:28.847020 23958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:38:28.847118 23958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:28.884140 23958 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.3:46385
I20260924 11:38:28.884315 24085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.3:46385 every 8 connection(s)
I20260924 11:38:28.885820 23958 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
I20260924 11:38:28.891526 24088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:28.889344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.3:57583)
0924 11:38:28.890205 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:28.890500 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":336,"threads_started":1}
W20260924 11:38:28.892284 24086 heartbeater.cc:644] Failed to heartbeat to 127.6.212.62:36825 (0 consecutive failures): Network error: Failed to ping master at 127.6.212.62:36825: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
I20260924 11:38:28.893247 24088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:28.892861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.3:37989)
0924 11:38:28.893042 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:28.893120 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260924 11:38:28.894253 24088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:28.893839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.3:47551)
0924 11:38:28.894043 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:28.894131 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260924 11:38:28.894474 24086 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 11:38:28.895249 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 23958
I20260924 11:38:28.905929 24090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:28.904633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.1:43831)
0924 11:38:28.905382 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:28.905545 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":361,"threads_started":1}
I20260924 11:38:29.165421 24091 raft_consensus.cc:482] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:38:29.165797 24091 raft_consensus.cc:504] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8832bc897e9d4f2191318fcdba73bb9d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 46385 } } peers { permanent_uuid: "63e7d703fab14977bd19cb32ff984f5e" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36853 } } peers { permanent_uuid: "62c2c7c8728542e5a9f8f9851794a98d" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46119 } }
I20260924 11:38:29.167618 24091 leader_election.cc:288] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8832bc897e9d4f2191318fcdba73bb9d (127.6.212.3:46385), 63e7d703fab14977bd19cb32ff984f5e (127.6.212.2:36853)
Sep 24 11:38:29 dist-test-slave-cq7c krb5kdc[22504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249907, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for kudu/127.6.212.2@KRBTEST.COM
Sep 24 11:38:29 dist-test-slave-cq7c krb5kdc[22504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249907, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for kudu/127.6.212.3@KRBTEST.COM
I20260924 11:38:29.195636 24094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:29.168036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.3:46385 (local address 127.6.212.1:53419)
0924 11:38:29.169969 (+ 1933us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:29.169985 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:38:29.170108 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:38:29.173107 (+ 2999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:38:29.173113 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:38:29.173800 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:38:29.174941 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:29.174975 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:29.177691 (+ 2716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:29.177698 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:29.180757 (+ 3059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:29.180784 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:29.182909 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:29.182914 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:29.183085 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:29.184014 (+ 929us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:38:29.184033 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:38:29.187069 (+ 3036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:38:29.191929 (+ 4860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:38:29.191940 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:38:29.191943 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:38:29.192338 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:38:29.192805 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:38:29.192814 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:38:29.192817 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:38:29.192940 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:38:29.193811 (+ 871us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:38:29.193815 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:38:29.194172 (+ 357us) client_negotiation.cc:769] Sending connection context
0924 11:38:29.194511 (+ 339us) client_negotiation.cc:240] Negotiation successful
0924 11:38:29.195137 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1778,"mutex_wait_us":936,"spinlock_wait_cycles":36864,"thread_start_us":1010,"threads_started":1}
I20260924 11:38:29.195796 24090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:29.168010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.2:36853 (local address 127.6.212.1:33999)
0924 11:38:29.168245 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:29.168306 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 11:38:29.168881 (+ 575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:38:29.172546 (+ 3665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:38:29.172559 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:38:29.173702 (+ 1143us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:38:29.174941 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:29.174975 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:29.177691 (+ 2716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:29.177698 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:29.180757 (+ 3059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:29.180784 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:29.182618 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:29.182629 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:29.182974 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:29.183995 (+ 1021us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:38:29.184038 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:38:29.186892 (+ 2854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:38:29.190979 (+ 4087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:38:29.191001 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:38:29.191026 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:38:29.191827 (+ 801us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:38:29.192395 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:38:29.192400 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:38:29.192403 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:38:29.192613 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:38:29.193703 (+ 1090us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:38:29.193712 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:38:29.194172 (+ 460us) client_negotiation.cc:769] Sending connection context
0924 11:38:29.194511 (+ 339us) client_negotiation.cc:240] Negotiation successful
0924 11:38:29.195315 (+ 804us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"mutex_wait_us":613,"spinlock_wait_cycles":22912}
I20260924 11:38:29.195880 24093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:29.168385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:53419 (local address 127.6.212.3:46385)
0924 11:38:29.169581 (+ 1196us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:29.169594 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:29.170099 (+ 505us) server_negotiation.cc:407] Connection header received
0924 11:38:29.172290 (+ 2191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:29.172305 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:29.172776 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:29.173111 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:38:29.175307 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:29.177376 (+ 2069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:29.181031 (+ 3655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:29.182289 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:29.182521 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:29.187654 (+ 5133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:38:29.187663 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:38:29.187678 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:38:29.187725 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:38:29.191728 (+ 4003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:29.192487 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:29.192503 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:29.192508 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:29.192677 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:29.193053 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:29.193058 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:29.193063 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:29.193549 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:38:29.193744 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:29.195020 (+ 1276us) server_negotiation.cc:299] Negotiation successful
0924 11:38:29.195463 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1010,"thread_start_us":852,"threads_started":1}
I20260924 11:38:29.196177 24092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:29.168315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:33999 (local address 127.6.212.2:36853)
0924 11:38:29.168964 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:29.168979 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:29.169026 (+ 47us) server_negotiation.cc:407] Connection header received
0924 11:38:29.171429 (+ 2403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:29.171444 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:29.172011 (+ 567us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:29.172405 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:38:29.175250 (+ 2845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:29.177376 (+ 2126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:29.181030 (+ 3654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:29.182289 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:29.182521 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:29.187341 (+ 4820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:38:29.187352 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:38:29.187409 (+ 57us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:38:29.187467 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:38:29.190742 (+ 3275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:29.192021 (+ 1279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:29.192030 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:29.192036 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:29.192241 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:29.192756 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:29.192761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:29.192765 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:29.193478 (+ 713us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:38:29.193691 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:29.194811 (+ 1120us) server_negotiation.cc:299] Negotiation successful
0924 11:38:29.195296 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":519,"threads_started":1}
W20260924 11:38:29.200125 23711 leader_election.cc:341] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8832bc897e9d4f2191318fcdba73bb9d (127.6.212.3:46385): Not found: Tablet not found: dfa84b724f4e49b782dd756fe1019bdc
W20260924 11:38:29.200580 23712 leader_election.cc:341] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 63e7d703fab14977bd19cb32ff984f5e (127.6.212.2:36853): Not found: Tablet not found: dfa84b724f4e49b782dd756fe1019bdc
I20260924 11:38:29.200829 23712 leader_election.cc:302] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62c2c7c8728542e5a9f8f9851794a98d; no voters: 63e7d703fab14977bd19cb32ff984f5e, 8832bc897e9d4f2191318fcdba73bb9d
I20260924 11:38:29.201298 24091 raft_consensus.cc:2726] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 11:38:29.403718 24095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:29.402700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.2:37109)
0924 11:38:29.403249 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:29.403464 (+ 215us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":243,"threads_started":1}
I20260924 11:38:29.895993 24096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:29.895121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.3:42953)
0924 11:38:29.895673 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:29.895784 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":290,"threads_started":1}
I20260924 11:38:29.907756 24097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:29.906973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.1:45729)
0924 11:38:29.907496 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:29.907584 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":282,"threads_started":1}
I20260924 11:38:30.405462 24098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:30.404734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.2:39869)
0924 11:38:30.405210 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:30.405293 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":270,"threads_started":1}
I20260924 11:38:30.897785 24099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:30.897042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.3:50267)
0924 11:38:30.897510 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:30.897594 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":247,"threads_started":1}
I20260924 11:38:30.909785 24100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:30.908706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.1:34749)
0924 11:38:30.909371 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:30.909497 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":285,"threads_started":1}
I20260924 11:38:31.407490 24101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:31.406532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.2:36921)
0924 11:38:31.407075 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:31.407200 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":256,"threads_started":1}
I20260924 11:38:31.436715 24102 raft_consensus.cc:482] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:38:31.436968 24102 raft_consensus.cc:504] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8832bc897e9d4f2191318fcdba73bb9d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 46385 } } peers { permanent_uuid: "63e7d703fab14977bd19cb32ff984f5e" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36853 } } peers { permanent_uuid: "62c2c7c8728542e5a9f8f9851794a98d" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46119 } }
I20260924 11:38:31.437776 24102 leader_election.cc:288] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8832bc897e9d4f2191318fcdba73bb9d (127.6.212.3:46385), 63e7d703fab14977bd19cb32ff984f5e (127.6.212.2:36853)
W20260924 11:38:31.438935 23712 leader_election.cc:341] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 63e7d703fab14977bd19cb32ff984f5e (127.6.212.2:36853): Not found: Tablet not found: dfa84b724f4e49b782dd756fe1019bdc
W20260924 11:38:31.439208 23711 leader_election.cc:341] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8832bc897e9d4f2191318fcdba73bb9d (127.6.212.3:46385): Not found: Tablet not found: dfa84b724f4e49b782dd756fe1019bdc
I20260924 11:38:31.439364 23711 leader_election.cc:302] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62c2c7c8728542e5a9f8f9851794a98d; no voters: 63e7d703fab14977bd19cb32ff984f5e, 8832bc897e9d4f2191318fcdba73bb9d
I20260924 11:38:31.439666 24102 raft_consensus.cc:2726] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 11:38:31.899637 24103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:31.898750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.3:42085)
0924 11:38:31.899296 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:31.899472 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":253,"threads_started":1}
I20260924 11:38:31.911824 24104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:31.910838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.1:58595)
0924 11:38:31.911481 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:31.911596 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":294,"threads_started":1}
I20260924 11:38:32.409773 24105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:32.408709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.2:49231)
0924 11:38:32.409293 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:32.409558 (+ 265us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":274,"threads_started":1}
I20260924 11:38:32.901479 24106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:32.900692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.3:54115)
0924 11:38:32.901208 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:32.901299 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":253,"threads_started":1}
I20260924 11:38:32.913652 24107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:32.912840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.1:58297)
0924 11:38:32.913327 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:32.913416 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":240,"threads_started":1}
I20260924 11:38:33.411769 24108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:33.410894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.2:46427)
0924 11:38:33.411492 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:33.411598 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":288,"threads_started":1}
I20260924 11:38:33.553890 24109 raft_consensus.cc:482] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:38:33.554132 24109 raft_consensus.cc:504] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8832bc897e9d4f2191318fcdba73bb9d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 46385 } } peers { permanent_uuid: "63e7d703fab14977bd19cb32ff984f5e" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36853 } } peers { permanent_uuid: "62c2c7c8728542e5a9f8f9851794a98d" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46119 } }
I20260924 11:38:33.554889 24109 leader_election.cc:288] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8832bc897e9d4f2191318fcdba73bb9d (127.6.212.3:46385), 63e7d703fab14977bd19cb32ff984f5e (127.6.212.2:36853)
W20260924 11:38:33.556066 23712 leader_election.cc:341] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 63e7d703fab14977bd19cb32ff984f5e (127.6.212.2:36853): Not found: Tablet not found: dfa84b724f4e49b782dd756fe1019bdc
W20260924 11:38:33.556267 23711 leader_election.cc:341] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8832bc897e9d4f2191318fcdba73bb9d (127.6.212.3:46385): Not found: Tablet not found: dfa84b724f4e49b782dd756fe1019bdc
I20260924 11:38:33.556401 23711 leader_election.cc:302] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62c2c7c8728542e5a9f8f9851794a98d; no voters: 63e7d703fab14977bd19cb32ff984f5e, 8832bc897e9d4f2191318fcdba73bb9d
I20260924 11:38:33.556674 24109 raft_consensus.cc:2726] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 11:38:33.903483 24110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:33.902562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.3:60239)
0924 11:38:33.903129 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:33.903239 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":282,"threads_started":1}
I20260924 11:38:33.915710 24111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:33.914810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.1:59989)
0924 11:38:33.915406 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:33.915505 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":331,"threads_started":1}
I20260924 11:38:34.413643 24112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:34.412835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.2:50951)
0924 11:38:34.413355 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:34.413443 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":273,"threads_started":1}
I20260924 11:38:34.905365 24113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:34.904505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.3:59719)
0924 11:38:34.905087 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:34.905180 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":305,"threads_started":1}
I20260924 11:38:34.917662 24114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:34.916811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.1:44889)
0924 11:38:34.917327 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:34.917455 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":280,"threads_started":1}
I20260924 11:38:35.415700 24115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:35.414757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.2:52771)
0924 11:38:35.415324 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:35.415486 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":265,"threads_started":1}
I20260924 11:38:35.907639 24116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:35.906522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.3:59377)
0924 11:38:35.907151 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:35.907299 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":277,"threads_started":1}
I20260924 11:38:35.919591 24117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:35.918617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.1:43349)
0924 11:38:35.919171 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:35.919277 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":239,"threads_started":1}
I20260924 11:38:36.417685 24118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:36.416851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.2:37269)
0924 11:38:36.417387 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:36.417471 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":304,"threads_started":1}
I20260924 11:38:36.909812 24119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:36.908864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.3:45647)
0924 11:38:36.909400 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:36.909486 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":318,"threads_started":1}
I20260924 11:38:36.921496 24120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:36.920715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.1:60937)
0924 11:38:36.921229 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:36.921321 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":256,"threads_started":1}
I20260924 11:38:36.946025 24121 raft_consensus.cc:482] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:38:36.946321 24121 raft_consensus.cc:504] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8832bc897e9d4f2191318fcdba73bb9d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 46385 } } peers { permanent_uuid: "63e7d703fab14977bd19cb32ff984f5e" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 36853 } } peers { permanent_uuid: "62c2c7c8728542e5a9f8f9851794a98d" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 46119 } }
I20260924 11:38:36.947103 24121 leader_election.cc:288] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8832bc897e9d4f2191318fcdba73bb9d (127.6.212.3:46385), 63e7d703fab14977bd19cb32ff984f5e (127.6.212.2:36853)
W20260924 11:38:36.948148 23711 leader_election.cc:341] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8832bc897e9d4f2191318fcdba73bb9d (127.6.212.3:46385): Not found: Tablet not found: dfa84b724f4e49b782dd756fe1019bdc
W20260924 11:38:36.948351 23712 leader_election.cc:341] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 63e7d703fab14977bd19cb32ff984f5e (127.6.212.2:36853): Not found: Tablet not found: dfa84b724f4e49b782dd756fe1019bdc
I20260924 11:38:36.948446 23712 leader_election.cc:302] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62c2c7c8728542e5a9f8f9851794a98d; no voters: 63e7d703fab14977bd19cb32ff984f5e, 8832bc897e9d4f2191318fcdba73bb9d
I20260924 11:38:36.948664 24121 raft_consensus.cc:2726] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 11:38:37.419605 24122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:37.418659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.2:42865)
0924 11:38:37.419269 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:37.419437 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":362,"threads_started":1}
I20260924 11:38:37.911615 24123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:37.910737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.3:52243)
0924 11:38:37.911277 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:37.911449 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":237,"threads_started":1}
I20260924 11:38:37.923674 24124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:37.922759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.1:36997)
0924 11:38:37.923350 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:37.923505 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":393,"threads_started":1}
I20260924 11:38:38.421363 24125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:38.420556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.2:47485)
0924 11:38:38.421022 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:38.421166 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":248,"threads_started":1}
I20260924 11:38:38.913575 24126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:38.912716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.3:33537)
0924 11:38:38.913265 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:38.913357 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":323,"threads_started":1}
I20260924 11:38:38.925565 24127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:38.924609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.1:36811)
0924 11:38:38.925185 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:38.925310 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":267,"threads_started":1}
I20260924 11:38:39.423295 24128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:39.422470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.2:40067)
0924 11:38:39.422981 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:39.423104 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":269,"threads_started":1}
I20260924 11:38:39.915333 24129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:39.914549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.3:59577)
0924 11:38:39.915088 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:39.915172 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":230,"threads_started":1}
I20260924 11:38:39.927569 24130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:39.926545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.1:39397)
0924 11:38:39.927210 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:39.927297 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":359,"threads_started":1}
I20260924 11:38:40.425083 24131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:40.424279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.2:58631)
0924 11:38:40.424743 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:40.424824 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":255,"threads_started":1}
I20260924 11:38:40.917192 24132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:40.916461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.3:51153)
0924 11:38:40.916929 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:40.917021 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":234,"threads_started":1}
I20260924 11:38:40.929147 24133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:40.928463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.1:55709)
0924 11:38:40.928881 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:40.928979 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":218,"threads_started":1}
I20260924 11:38:41.427172 24134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:41.426231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.2:42945)
0924 11:38:41.426890 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:41.426986 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":361,"threads_started":1}
I20260924 11:38:41.919045 24135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:41.918291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.3:54981)
0924 11:38:41.918779 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:41.918871 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":220,"threads_started":1}
I20260924 11:38:41.930717 24136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:41.930000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.1:55177)
0924 11:38:41.930445 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:41.930536 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":244,"threads_started":1}
I20260924 11:38:42.428893 24137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:42.428159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.2:47065)
0924 11:38:42.428625 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:42.428702 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":220,"threads_started":1}
I20260924 11:38:42.921104 24138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:42.920202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.3:34697)
0924 11:38:42.920694 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:42.920787 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":256,"threads_started":1}
I20260924 11:38:42.932746 24139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:42.931975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.1:57977)
0924 11:38:42.932433 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:42.932570 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":225,"threads_started":1}
I20260924 11:38:43.430759 24140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:43.430026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.2:55479)
0924 11:38:43.430492 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:43.430594 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":219,"threads_started":1}
I20260924 11:38:43.906838 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 23653
I20260924 11:38:43.923345 24142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:43.922367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36825 (local address 127.6.212.3:53541)
0924 11:38:43.923008 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:43.923161 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.62:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":259,"threads_started":1}
I20260924 11:38:43.923332 24141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:43.908165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53446 (local address 127.6.212.1:46119)
0924 11:38:43.908723 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:43.908731 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:43.908759 (+ 28us) server_negotiation.cc:407] Connection header received
0924 11:38:43.908849 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:43.908860 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:43.909163 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:43.909353 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:38:43.910956 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:43.912055 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:43.913240 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:43.913804 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:43.913902 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:43.916689 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:38:43.916702 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:38:43.916718 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:38:43.916756 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:38:43.920312 (+ 3556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:43.921051 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:43.921059 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:43.921062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:43.921203 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:43.921624 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:43.921628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:43.921630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:43.922053 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:38:43.922405 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:43.922667 (+ 262us) server_negotiation.cc:299] Negotiation successful
0924 11:38:43.922938 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":255,"threads_started":1}
I20260924 11:38:43.924870 23819 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:38:44.016429 24144 raft_consensus.cc:482] T dfa84b724f4e49b782dd756fe1019bdc P 62c2c7c8728542e5a9f8f9851794a98d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:38:44.016511 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 23653
I20260924 11:38:44.037144 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 23827
I20260924 11:38:44.051271 24145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:44.037972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33880 (local address 127.6.212.2:36853)
0924 11:38:44.038464 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:44.038468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:44.038485 (+ 17us) server_negotiation.cc:407] Connection header received
0924 11:38:44.038540 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:44.038543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:44.038603 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:44.038700 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:38:44.040297 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:44.041155 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:44.042486 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:44.042904 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:44.043013 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:44.045735 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:38:44.045740 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:38:44.045743 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:38:44.045778 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:38:44.048626 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:44.049279 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:44.049283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:44.049285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:44.049430 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:44.049910 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:44.049914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:44.049916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:44.050212 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:38:44.050422 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:44.050790 (+ 368us) server_negotiation.cc:299] Negotiation successful
0924 11:38:44.050979 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":282,"threads_started":1}
I20260924 11:38:44.052002 23950 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:38:44.141639 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 23827
I20260924 11:38:44.160858 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 23958
I20260924 11:38:44.176095 24147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:44.161760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56192 (local address 127.6.212.3:46385)
0924 11:38:44.162293 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:44.162298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:44.162316 (+ 18us) server_negotiation.cc:407] Connection header received
0924 11:38:44.162374 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:44.162377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:44.162495 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:44.162646 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:38:44.164258 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:44.165294 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:44.166671 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:44.167091 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:44.167238 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:44.169998 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:38:44.170005 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:38:44.170010 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:38:44.170049 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:38:44.173248 (+ 3199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:44.173975 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:44.173980 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:44.173986 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:44.174125 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:44.174561 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:44.174565 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:44.174567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:44.174826 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:38:44.175109 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:44.175454 (+ 345us) server_negotiation.cc:299] Negotiation successful
0924 11:38:44.175676 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":280,"threads_started":1}
I20260924 11:38:44.176802 24081 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:38:44.262511 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 23958
2026-09-24T11:38:44Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65035 ms)
[----------] 30 tests from SecurityITest (307163 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:38:44 dist-test-slave-cq7c krb5kdc[24151](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:38:44 dist-test-slave-cq7c krb5kdc[24151](info): set up 2 sockets
Sep 24 11:38:44 dist-test-slave-cq7c krb5kdc[24151](info): commencing operation
krb5kdc: starting...
W20260924 11:38:45.391613 6992 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.002s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Sep 24 11:38:45 dist-test-slave-cq7c krb5kdc[24151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:38:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:38:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.212.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
I20260924 11:38:45.672205 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:46353
--webserver_interface=127.6.212.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:46353
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:38:46.033890 24164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:38:46.034230 24164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:38:46.034322 24164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:38:46.041956 24164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:38:46.042084 24164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:38:46.042137 24164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:38:46.042188 24164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:38:46.042233 24164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:38:46.052636 24164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:46353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:46353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.212.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:38:46.053982 24164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:38:46.055250 24164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:38:46.067476 24169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:38:46.067576 24172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:38:46.067723 24170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:38:46.068642 24164 server_base.cc:1034] running on GCE node
I20260924 11:38:46.069567 24164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:38:46.071132 24164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:38:46.072360 24164 hybrid_clock.cc:648] HybridClock initialized: now 1790249926072298 us; error 85 us; skew 500 ppm
I20260924 11:38:46.072731 24164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:38:46 dist-test-slave-cq7c krb5kdc[24151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249926, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:38:46.081321 24164 init.cc:377] Logged in from keytab as kudu/127.6.212.62@KRBTEST.COM (short username kudu)
I20260924 11:38:46.083032 24164 webserver.cc:460] Webserver started at http://127.6.212.62:45859/ using document root <none> and password file <none>
I20260924 11:38:46.083822 24164 fs_manager.cc:359] Metadata directory not provided
I20260924 11:38:46.083933 24164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:38:46.084260 24164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:38:46.087699 24164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "df84b80e0fc74ce692c6f00e60e2886a"
format_stamp: "Formatted at 2026-09-24 11:38:46 on dist-test-slave-cq7c"
server_key: "e040f26c0d0a38606d0b9c46ed7747d4"
server_key_iv: "a6fd6c3383f071e9da7c1204dddf59cf"
server_key_version: "encryptionkey@0"
I20260924 11:38:46.088513 24164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "df84b80e0fc74ce692c6f00e60e2886a"
format_stamp: "Formatted at 2026-09-24 11:38:46 on dist-test-slave-cq7c"
server_key: "e040f26c0d0a38606d0b9c46ed7747d4"
server_key_iv: "a6fd6c3383f071e9da7c1204dddf59cf"
server_key_version: "encryptionkey@0"
I20260924 11:38:46.094163 24164 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 11:38:46.098167 24179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:46.099798 24164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 11:38:46.100001 24164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "df84b80e0fc74ce692c6f00e60e2886a"
format_stamp: "Formatted at 2026-09-24 11:38:46 on dist-test-slave-cq7c"
server_key: "e040f26c0d0a38606d0b9c46ed7747d4"
server_key_iv: "a6fd6c3383f071e9da7c1204dddf59cf"
server_key_version: "encryptionkey@0"
I20260924 11:38:46.100226 24164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:38:46.115873 24164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:38:46.121536 24164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:38:46.121778 24164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:38:46.141012 24164 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.62:46353
I20260924 11:38:46.141145 24230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.62:46353 every 8 connection(s)
I20260924 11:38:46.142488 24164 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:38:46.146548 24231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:38:46.149220 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 24164
I20260924 11:38:46.149442 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:38:46.150252 6992 external_mini_cluster.cc:1428] Setting key ca6ad8462720124a4721b66cc75d6dfe
I20260924 11:38:46.159709 24231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df84b80e0fc74ce692c6f00e60e2886a: Bootstrap starting.
I20260924 11:38:46.163671 24231 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df84b80e0fc74ce692c6f00e60e2886a: Neither blocks nor log segments found. Creating new log.
I20260924 11:38:46.165081 24231 log.cc:824] T 00000000000000000000000000000000 P df84b80e0fc74ce692c6f00e60e2886a: Log is configured to *not* fsync() on all Append() calls
Sep 24 11:38:46 dist-test-slave-cq7c krb5kdc[24151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:38:46.168361 24231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df84b80e0fc74ce692c6f00e60e2886a: No bootstrap required, opened a new log
I20260924 11:38:46.174665 24234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:46.152437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35360 (local address 127.6.212.62:46353)
0924 11:38:46.153333 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:46.153345 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:46.153387 (+ 42us) server_negotiation.cc:407] Connection header received
0924 11:38:46.155053 (+ 1666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:46.155075 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:46.155548 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:46.155879 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:38:46.157695 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:46.159067 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:46.160959 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:46.161641 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:46.161774 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:46.167101 (+ 5327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:38:46.167109 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:38:46.167125 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:38:46.167159 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:38:46.170374 (+ 3215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:46.171234 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:46.171241 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:46.171247 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:46.171500 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:46.171904 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:46.171907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:46.171909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:46.172442 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:38:46.172667 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:46.173169 (+ 502us) server_negotiation.cc:299] Negotiation successful
0924 11:38:46.173596 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":479,"threads_started":1}
I20260924 11:38:46.177475 24231 raft_consensus.cc:348] T 00000000000000000000000000000000 P df84b80e0fc74ce692c6f00e60e2886a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df84b80e0fc74ce692c6f00e60e2886a" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 46353 } }
I20260924 11:38:46.177834 24231 raft_consensus.cc:374] T 00000000000000000000000000000000 P df84b80e0fc74ce692c6f00e60e2886a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:38:46.177944 24231 raft_consensus.cc:729] T 00000000000000000000000000000000 P df84b80e0fc74ce692c6f00e60e2886a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df84b80e0fc74ce692c6f00e60e2886a, State: Initialized, Role: FOLLOWER
I20260924 11:38:46.178483 24231 consensus_queue.cc:260] T 00000000000000000000000000000000 P df84b80e0fc74ce692c6f00e60e2886a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df84b80e0fc74ce692c6f00e60e2886a" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 46353 } }
I20260924 11:38:46.178715 24231 raft_consensus.cc:388] T 00000000000000000000000000000000 P df84b80e0fc74ce692c6f00e60e2886a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:38:46.178857 24231 raft_consensus.cc:482] T 00000000000000000000000000000000 P df84b80e0fc74ce692c6f00e60e2886a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:38:46.179028 24231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df84b80e0fc74ce692c6f00e60e2886a [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:38:46.181295 24231 raft_consensus.cc:504] T 00000000000000000000000000000000 P df84b80e0fc74ce692c6f00e60e2886a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df84b80e0fc74ce692c6f00e60e2886a" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 46353 } }
I20260924 11:38:46.181784 24231 leader_election.cc:302] T 00000000000000000000000000000000 P df84b80e0fc74ce692c6f00e60e2886a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df84b80e0fc74ce692c6f00e60e2886a; no voters:
I20260924 11:38:46.182361 24231 leader_election.cc:288] T 00000000000000000000000000000000 P df84b80e0fc74ce692c6f00e60e2886a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:38:46.182562 24236 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df84b80e0fc74ce692c6f00e60e2886a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:38:46.183789 24236 raft_consensus.cc:686] T 00000000000000000000000000000000 P df84b80e0fc74ce692c6f00e60e2886a [term 1 LEADER]: Becoming Leader. State: Replica: df84b80e0fc74ce692c6f00e60e2886a, State: Running, Role: LEADER
I20260924 11:38:46.184662 24231 sys_catalog.cc:564] T 00000000000000000000000000000000 P df84b80e0fc74ce692c6f00e60e2886a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:38:46.184495 24236 consensus_queue.cc:237] T 00000000000000000000000000000000 P df84b80e0fc74ce692c6f00e60e2886a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df84b80e0fc74ce692c6f00e60e2886a" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 46353 } }
I20260924 11:38:46.187498 24238 sys_catalog.cc:455] T 00000000000000000000000000000000 P df84b80e0fc74ce692c6f00e60e2886a [sys.catalog]: SysCatalogTable state changed. Reason: New leader df84b80e0fc74ce692c6f00e60e2886a. Latest consensus state: current_term: 1 leader_uuid: "df84b80e0fc74ce692c6f00e60e2886a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df84b80e0fc74ce692c6f00e60e2886a" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 46353 } } }
I20260924 11:38:46.187953 24238 sys_catalog.cc:458] T 00000000000000000000000000000000 P df84b80e0fc74ce692c6f00e60e2886a [sys.catalog]: This master's current role is: LEADER
I20260924 11:38:46.187783 24237 sys_catalog.cc:455] T 00000000000000000000000000000000 P df84b80e0fc74ce692c6f00e60e2886a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df84b80e0fc74ce692c6f00e60e2886a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df84b80e0fc74ce692c6f00e60e2886a" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 46353 } } }
I20260924 11:38:46.188158 24237 sys_catalog.cc:458] T 00000000000000000000000000000000 P df84b80e0fc74ce692c6f00e60e2886a [sys.catalog]: This master's current role is: LEADER
I20260924 11:38:46.191789 24244 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:38:46.197316 24244 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:38:46.206470 24244 catalog_manager.cc:1347] Generated new cluster ID: 8633cdd40ce34a49bc839ba9626191a7
I20260924 11:38:46.206600 24244 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:38:46.222009 24244 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:38:46.225071 24244 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:38:46.239333 24244 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df84b80e0fc74ce692c6f00e60e2886a: Generated new TSK 0
I20260924 11:38:46.240566 24244 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.212.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab.
Entry for principal kudu/127.6.212.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab.
I20260924 11:38:46.323863 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.1:0
--local_ip_for_outbound_sockets=127.6.212.1
--webserver_interface=127.6.212.1
--webserver_port=0
--tserver_master_addrs=127.6.212.62:46353
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab
--principal=kudu/127.6.212.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:38:46.663146 24257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:38:46.663487 24257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:38:46.663589 24257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:38:46.670979 24257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:38:46.671094 24257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:38:46.671208 24257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.1
I20260924 11:38:46.681345 24257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab
--principal=kudu/127.6.212.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.212.1
--webserver_port=0
--tserver_master_addrs=127.6.212.62:46353
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.1
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:38:46.682631 24257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:38:46.683988 24257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:38:46.698218 24265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:38:46.698374 24263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:38:46.698197 24262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:38:46.699662 24257 server_base.cc:1034] running on GCE node
I20260924 11:38:46.700434 24257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:38:46.701427 24257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:38:46.702656 24257 hybrid_clock.cc:648] HybridClock initialized: now 1790249926702606 us; error 32 us; skew 500 ppm
I20260924 11:38:46.703030 24257 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:38:46 dist-test-slave-cq7c krb5kdc[24151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249926, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:38:46.706646 24257 init.cc:377] Logged in from keytab as kudu/127.6.212.1@KRBTEST.COM (short username kudu)
I20260924 11:38:46.708334 24257 webserver.cc:460] Webserver started at http://127.6.212.1:44713/ using document root <none> and password file <none>
I20260924 11:38:46.709065 24257 fs_manager.cc:359] Metadata directory not provided
I20260924 11:38:46.709209 24257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:38:46.709530 24257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:38:46.712644 24257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-0/data/instance:
uuid: "19cac7fb427c4a118d0c7e86bf622b00"
format_stamp: "Formatted at 2026-09-24 11:38:46 on dist-test-slave-cq7c"
server_key: "daa2fd7b29da7eff95c1819d24744074"
server_key_iv: "db839725f4305ea6686ae6501e323dd3"
server_key_version: "encryptionkey@0"
I20260924 11:38:46.713501 24257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance:
uuid: "19cac7fb427c4a118d0c7e86bf622b00"
format_stamp: "Formatted at 2026-09-24 11:38:46 on dist-test-slave-cq7c"
server_key: "daa2fd7b29da7eff95c1819d24744074"
server_key_iv: "db839725f4305ea6686ae6501e323dd3"
server_key_version: "encryptionkey@0"
I20260924 11:38:46.718869 24257 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260924 11:38:46.722427 24272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:46.723899 24257 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:38:46.724068 24257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-0/wal
uuid: "19cac7fb427c4a118d0c7e86bf622b00"
format_stamp: "Formatted at 2026-09-24 11:38:46 on dist-test-slave-cq7c"
server_key: "daa2fd7b29da7eff95c1819d24744074"
server_key_iv: "db839725f4305ea6686ae6501e323dd3"
server_key_version: "encryptionkey@0"
I20260924 11:38:46.724287 24257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:38:46.745682 24257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:38:46.750171 24257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:38:46.750514 24257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:38:46.751581 24257 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:38:46.752974 24257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:38:46.753085 24257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:46.753239 24257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:38:46.753315 24257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:46.789165 24257 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.1:34557
I20260924 11:38:46.789248 24384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.1:34557 every 8 connection(s)
I20260924 11:38:46.790566 24257 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
I20260924 11:38:46.800259 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 24257
I20260924 11:38:46.800469 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance
I20260924 11:38:46.801118 6992 external_mini_cluster.cc:1428] Setting key f088d75103f054d5bfebabb70e5e6a5e
Sep 24 11:38:46 dist-test-slave-cq7c krb5kdc[24151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249926, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:38:46.812770 24387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:46.793454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:48023 (local address 127.6.212.62:46353)
0924 11:38:46.793941 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:46.793946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:46.794775 (+ 829us) server_negotiation.cc:407] Connection header received
0924 11:38:46.796943 (+ 2168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:46.796949 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:46.797056 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:46.797179 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:38:46.798819 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:46.799699 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:46.801536 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:46.802001 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:46.802147 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:46.806515 (+ 4368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:38:46.806521 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:38:46.806525 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:38:46.806563 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:38:46.809388 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:46.810206 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:46.810213 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:46.810217 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:46.810352 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:46.810835 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:46.810845 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:46.810847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:46.811123 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:38:46.811286 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:46.812095 (+ 809us) server_negotiation.cc:299] Negotiation successful
0924 11:38:46.812358 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":268,"threads_started":1}
I20260924 11:38:46.813843 24388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:46.793887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:46353 (local address 127.6.212.1:48023)
0924 11:38:46.794605 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:46.794654 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 11:38:46.796653 (+ 1999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:38:46.797448 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:38:46.797457 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:38:46.798013 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:38:46.798638 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:46.798658 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:46.799866 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:46.799870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:46.801385 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:46.801397 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:46.802259 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:46.802269 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:46.802532 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:46.803280 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:38:46.803318 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:38:46.806251 (+ 2933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:38:46.809559 (+ 3308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:38:46.809569 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:38:46.809589 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:38:46.810076 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:38:46.810479 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:38:46.810485 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:38:46.810489 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:38:46.810698 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:38:46.811255 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:38:46.811263 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:38:46.811703 (+ 440us) client_negotiation.cc:769] Sending connection context
0924 11:38:46.811995 (+ 292us) client_negotiation.cc:240] Negotiation successful
0924 11:38:46.812611 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":268,"threads_started":1}
I20260924 11:38:46.815574 24385 heartbeater.cc:347] Connected to a master server at 127.6.212.62:46353
I20260924 11:38:46.815883 24385 heartbeater.cc:464] Registering TS with master...
I20260924 11:38:46.816623 24385 heartbeater.cc:511] Master 127.6.212.62:46353 requested a full tablet report, sending...
I20260924 11:38:46.819057 24196 ts_manager.cc:194] Registered new tserver with Master: 19cac7fb427c4a118d0c7e86bf622b00 (127.6.212.1:34557)
I20260924 11:38:46.821309 24196 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.1@KRBTEST.COM'} at 127.6.212.1:48023
WARNING: no policy specified for kudu/127.6.212.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab.
Entry for principal kudu/127.6.212.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab.
I20260924 11:38:46.862715 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.2:0
--local_ip_for_outbound_sockets=127.6.212.2
--webserver_interface=127.6.212.2
--webserver_port=0
--tserver_master_addrs=127.6.212.62:46353
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab
--principal=kudu/127.6.212.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:38:47.206106 24391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:38:47.206389 24391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:38:47.206485 24391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:38:47.213927 24391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:38:47.214051 24391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:38:47.214165 24391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.2
I20260924 11:38:47.224332 24391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab
--principal=kudu/127.6.212.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.212.2
--webserver_port=0
--tserver_master_addrs=127.6.212.62:46353
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.2
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:38:47.225594 24391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:38:47.227111 24391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:38:47.241091 24396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:38:47.241254 24399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:38:47.241290 24397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:38:47.242370 24391 server_base.cc:1034] running on GCE node
I20260924 11:38:47.243146 24391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:38:47.244273 24391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:38:47.245525 24391 hybrid_clock.cc:648] HybridClock initialized: now 1790249927245448 us; error 85 us; skew 500 ppm
I20260924 11:38:47.245882 24391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:38:47 dist-test-slave-cq7c krb5kdc[24151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249927, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:38:47.249926 24391 init.cc:377] Logged in from keytab as kudu/127.6.212.2@KRBTEST.COM (short username kudu)
I20260924 11:38:47.251597 24391 webserver.cc:460] Webserver started at http://127.6.212.2:35215/ using document root <none> and password file <none>
I20260924 11:38:47.252348 24391 fs_manager.cc:359] Metadata directory not provided
I20260924 11:38:47.252504 24391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:38:47.252831 24391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:38:47.256124 24391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-1/data/instance:
uuid: "56e40517e1654c698471e47ed9fba4f9"
format_stamp: "Formatted at 2026-09-24 11:38:47 on dist-test-slave-cq7c"
server_key: "d7c415a94c244727a693d8627c5a22e6"
server_key_iv: "7378e3ca8ea6cd31d6f13cd5bda9f3b4"
server_key_version: "encryptionkey@0"
I20260924 11:38:47.256982 24391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance:
uuid: "56e40517e1654c698471e47ed9fba4f9"
format_stamp: "Formatted at 2026-09-24 11:38:47 on dist-test-slave-cq7c"
server_key: "d7c415a94c244727a693d8627c5a22e6"
server_key_iv: "7378e3ca8ea6cd31d6f13cd5bda9f3b4"
server_key_version: "encryptionkey@0"
I20260924 11:38:47.262606 24391 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 11:38:47.266799 24406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:47.268293 24391 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:38:47.268464 24391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-1/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-1/wal
uuid: "56e40517e1654c698471e47ed9fba4f9"
format_stamp: "Formatted at 2026-09-24 11:38:47 on dist-test-slave-cq7c"
server_key: "d7c415a94c244727a693d8627c5a22e6"
server_key_iv: "7378e3ca8ea6cd31d6f13cd5bda9f3b4"
server_key_version: "encryptionkey@0"
I20260924 11:38:47.268667 24391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:38:47.291698 24391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:38:47.295950 24391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:38:47.296234 24391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:38:47.297200 24391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:38:47.298532 24391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:38:47.298637 24391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:47.298779 24391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:38:47.298862 24391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:47.334676 24391 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.2:46589
I20260924 11:38:47.334733 24518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.2:46589 every 8 connection(s)
I20260924 11:38:47.336176 24391 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
I20260924 11:38:47.339531 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 24391
I20260924 11:38:47.339748 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance
I20260924 11:38:47.340278 6992 external_mini_cluster.cc:1428] Setting key fdee3f83660e6d0d8cb9f248567008cc
Sep 24 11:38:47 dist-test-slave-cq7c krb5kdc[24151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249927, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.2@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:38:47.361748 24521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:47.339008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:40533 (local address 127.6.212.62:46353)
0924 11:38:47.339632 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:47.339640 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:47.340753 (+ 1113us) server_negotiation.cc:407] Connection header received
0924 11:38:47.343260 (+ 2507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:47.343264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:47.343428 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:47.343525 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:38:47.345839 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:47.346606 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:47.349197 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:47.349640 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:47.349741 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:47.354312 (+ 4571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:38:47.354321 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:38:47.354328 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:38:47.354374 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:38:47.358200 (+ 3826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:47.358962 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:47.358970 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:47.358976 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:47.359137 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:47.359651 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:47.359658 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:47.359664 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:47.360052 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:38:47.360202 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:47.360937 (+ 735us) server_negotiation.cc:299] Negotiation successful
0924 11:38:47.361238 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":354,"threads_started":1}
I20260924 11:38:47.362398 24522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:47.339588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:46353 (local address 127.6.212.2:40533)
0924 11:38:47.340545 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:47.340605 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 11:38:47.342919 (+ 2314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:38:47.343828 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:38:47.343842 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:38:47.344715 (+ 873us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:38:47.345621 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:47.345657 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:47.346763 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:47.346772 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:47.349034 (+ 2262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:47.349054 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:47.349895 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:47.349905 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:47.350260 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:47.351036 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:38:47.351074 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:38:47.354056 (+ 2982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:38:47.358350 (+ 4294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:38:47.358357 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:38:47.358373 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:38:47.358826 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:38:47.359249 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:38:47.359252 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:38:47.359254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:38:47.359519 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:38:47.360180 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:38:47.360186 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:38:47.360551 (+ 365us) client_negotiation.cc:769] Sending connection context
0924 11:38:47.360784 (+ 233us) client_negotiation.cc:240] Negotiation successful
0924 11:38:47.361216 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":407,"threads_started":1}
I20260924 11:38:47.364074 24519 heartbeater.cc:347] Connected to a master server at 127.6.212.62:46353
I20260924 11:38:47.364387 24519 heartbeater.cc:464] Registering TS with master...
I20260924 11:38:47.365114 24519 heartbeater.cc:511] Master 127.6.212.62:46353 requested a full tablet report, sending...
I20260924 11:38:47.366669 24196 ts_manager.cc:194] Registered new tserver with Master: 56e40517e1654c698471e47ed9fba4f9 (127.6.212.2:46589)
I20260924 11:38:47.367540 24196 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.2@KRBTEST.COM'} at 127.6.212.2:40533
WARNING: no policy specified for kudu/127.6.212.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab.
Entry for principal kudu/127.6.212.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab.
I20260924 11:38:47.402841 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.3:0
--local_ip_for_outbound_sockets=127.6.212.3
--webserver_interface=127.6.212.3
--webserver_port=0
--tserver_master_addrs=127.6.212.62:46353
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab
--principal=kudu/127.6.212.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:38:47.744398 24525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:38:47.744676 24525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:38:47.744771 24525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:38:47.752137 24525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:38:47.752249 24525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:38:47.752362 24525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.3
I20260924 11:38:47.762408 24525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab
--principal=kudu/127.6.212.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.212.3
--webserver_port=0
--tserver_master_addrs=127.6.212.62:46353
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.3
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:38:47.763692 24525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:38:47.764966 24525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:38:47.778216 24530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:38:47.778309 24533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:38:47.778256 24531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:38:47.779623 24525 server_base.cc:1034] running on GCE node
I20260924 11:38:47.780184 24525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:38:47.781083 24525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:38:47.782310 24525 hybrid_clock.cc:648] HybridClock initialized: now 1790249927782273 us; error 37 us; skew 500 ppm
I20260924 11:38:47.782635 24525 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:38:47 dist-test-slave-cq7c krb5kdc[24151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249927, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:38:47.786149 24525 init.cc:377] Logged in from keytab as kudu/127.6.212.3@KRBTEST.COM (short username kudu)
I20260924 11:38:47.787664 24525 webserver.cc:460] Webserver started at http://127.6.212.3:41707/ using document root <none> and password file <none>
I20260924 11:38:47.788386 24525 fs_manager.cc:359] Metadata directory not provided
I20260924 11:38:47.788522 24525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:38:47.788893 24525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:38:47.792155 24525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-2/data/instance:
uuid: "b775d8ccb23a433ebf09ca00459dc55a"
format_stamp: "Formatted at 2026-09-24 11:38:47 on dist-test-slave-cq7c"
server_key: "d8bd1d3dae567f2ddad6d61cf1e63136"
server_key_iv: "95aa78438f937c5e6c7925d9630ff76a"
server_key_version: "encryptionkey@0"
I20260924 11:38:47.793045 24525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance:
uuid: "b775d8ccb23a433ebf09ca00459dc55a"
format_stamp: "Formatted at 2026-09-24 11:38:47 on dist-test-slave-cq7c"
server_key: "d8bd1d3dae567f2ddad6d61cf1e63136"
server_key_iv: "95aa78438f937c5e6c7925d9630ff76a"
server_key_version: "encryptionkey@0"
I20260924 11:38:47.798590 24525 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 11:38:47.802383 24540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:47.804093 24525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:38:47.804272 24525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-2/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-2/wal
uuid: "b775d8ccb23a433ebf09ca00459dc55a"
format_stamp: "Formatted at 2026-09-24 11:38:47 on dist-test-slave-cq7c"
server_key: "d8bd1d3dae567f2ddad6d61cf1e63136"
server_key_iv: "95aa78438f937c5e6c7925d9630ff76a"
server_key_version: "encryptionkey@0"
I20260924 11:38:47.804495 24525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:38:47.821671 24525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:38:47.824713 24385 heartbeater.cc:503] Master 127.6.212.62:46353 was elected leader, sending a full tablet report...
I20260924 11:38:47.826666 24525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:38:47.826968 24525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:38:47.828104 24525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:38:47.829478 24525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:38:47.829550 24525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:47.829669 24525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:38:47.829715 24525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:47.863619 24525 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.3:43361
I20260924 11:38:47.863641 24652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.3:43361 every 8 connection(s)
I20260924 11:38:47.864965 24525 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
I20260924 11:38:47.869560 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 24525
I20260924 11:38:47.869758 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance
I20260924 11:38:47.870201 6992 external_mini_cluster.cc:1428] Setting key f2973717847c5507f0fcfc36dbcc1b1c
Sep 24 11:38:47 dist-test-slave-cq7c krb5kdc[24151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249927, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.3@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:38:47.885995 24655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:47.867666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:53559 (local address 127.6.212.62:46353)
0924 11:38:47.868123 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:47.868127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:47.869411 (+ 1284us) server_negotiation.cc:407] Connection header received
0924 11:38:47.871951 (+ 2540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:47.871956 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:47.872063 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:47.872173 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:38:47.874034 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:47.874808 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:47.876425 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:47.876872 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:47.876992 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:47.880393 (+ 3401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:38:47.880404 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:38:47.880407 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:38:47.880437 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:38:47.882915 (+ 2478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:47.883662 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:47.883667 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:47.883675 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:47.883796 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:47.884212 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:47.884217 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:47.884220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:47.884521 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:38:47.884676 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:47.885375 (+ 699us) server_negotiation.cc:299] Negotiation successful
0924 11:38:47.885605 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":223,"threads_started":1}
I20260924 11:38:47.886899 24656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:47.868347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:46353 (local address 127.6.212.3:53559)
0924 11:38:47.869221 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:47.869262 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 11:38:47.871654 (+ 2392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:38:47.872509 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:38:47.872526 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:38:47.873299 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:38:47.873882 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:47.873899 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:47.874968 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:47.874974 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:47.876273 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:47.876284 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:47.877102 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:47.877107 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:47.877315 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:47.877966 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:38:47.877993 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:38:47.880198 (+ 2205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:38:47.883046 (+ 2848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:38:47.883055 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:38:47.883071 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:38:47.883547 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:38:47.883905 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:38:47.883911 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:38:47.883915 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:38:47.884100 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:38:47.884642 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:38:47.884650 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:38:47.884971 (+ 321us) client_negotiation.cc:769] Sending connection context
0924 11:38:47.885246 (+ 275us) client_negotiation.cc:240] Negotiation successful
0924 11:38:47.885809 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":293,"threads_started":1}
I20260924 11:38:47.888397 24653 heartbeater.cc:347] Connected to a master server at 127.6.212.62:46353
I20260924 11:38:47.888723 24653 heartbeater.cc:464] Registering TS with master...
I20260924 11:38:47.889580 24653 heartbeater.cc:511] Master 127.6.212.62:46353 requested a full tablet report, sending...
I20260924 11:38:47.891281 24196 ts_manager.cc:194] Registered new tserver with Master: b775d8ccb23a433ebf09ca00459dc55a (127.6.212.3:43361)
I20260924 11:38:47.892179 24196 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.3@KRBTEST.COM'} at 127.6.212.3:53559
I20260924 11:38:47.897802 6992 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:38:47.915848 24655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:47.902734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37522 (local address 127.6.212.62:46353)
0924 11:38:47.902995 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:47.903016 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:47.903495 (+ 479us) server_negotiation.cc:407] Connection header received
0924 11:38:47.903672 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:47.903676 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:47.903736 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:47.903817 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:38:47.905479 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:47.906192 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:47.907733 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:47.908255 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:47.908392 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:47.910911 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:38:47.910918 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:38:47.910922 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:38:47.910962 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:38:47.913385 (+ 2423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:47.914124 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:47.914127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:47.914130 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:47.914243 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:47.914626 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:47.914630 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:47.914631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:47.914923 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:38:47.915098 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:47.915459 (+ 361us) server_negotiation.cc:299] Negotiation successful
0924 11:38:47.915625 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260924 11:38:47.920089 24196 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37522:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:38:47.927829 24196 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:38:47.955384 24667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:47.941550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:34557 (local address 127.0.0.1:50706)
0924 11:38:47.943443 (+ 1893us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:47.943479 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 11:38:47.943667 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:38:47.944812 (+ 1145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:38:47.944823 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:38:47.944864 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:38:47.945360 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:47.945376 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:47.947978 (+ 2602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:47.947986 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:47.951143 (+ 3157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:47.951165 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:47.954081 (+ 2916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:47.954090 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:47.954405 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:47.954482 (+ 77us) client_negotiation.cc:769] Sending connection context
0924 11:38:47.954645 (+ 163us) client_negotiation.cc:240] Negotiation successful
0924 11:38:47.954902 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1633,"threads_started":1}
I20260924 11:38:47.956177 24668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:47.941893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50706 (local address 127.6.212.1:34557)
0924 11:38:47.942695 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:47.942709 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:47.943595 (+ 886us) server_negotiation.cc:407] Connection header received
0924 11:38:47.943912 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:47.943924 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:47.944241 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:47.944472 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:38:47.945630 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:47.947766 (+ 2136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:47.951417 (+ 3651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:47.953431 (+ 2014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:47.953592 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:47.953897 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:47.955688 (+ 1791us) server_negotiation.cc:299] Negotiation successful
0924 11:38:47.955858 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":374,"threads_started":1}
I20260924 11:38:47.960562 24320 tablet_service.cc:1467] Processing CreateTablet for tablet e6a2b637256d4ffcb9199bea02454320 (DEFAULT_TABLE table=test-table [id=4976a2abf20c4b5aabb7cd62d1a7df27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:38:47.961951 24320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6a2b637256d4ffcb9199bea02454320. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:38:47.966017 24666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:47.941766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.2:46589 (local address 127.0.0.1:56300)
0924 11:38:47.955236 (+ 13470us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:47.955256 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 11:38:47.957613 (+ 2357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:38:47.958597 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:38:47.958601 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:38:47.958627 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:38:47.958946 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:47.958957 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:47.960803 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:47.960808 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:47.962896 (+ 2088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:47.962910 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:47.965174 (+ 2264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:47.965179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:47.965352 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:47.965384 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 11:38:47.965491 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 11:38:47.965631 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13334,"mutex_wait_us":33,"thread_start_us":1293,"threads_started":1}
I20260924 11:38:47.966298 24669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:47.942327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56300 (local address 127.6.212.2:46589)
0924 11:38:47.942879 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:47.942892 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:47.955389 (+ 12497us) server_negotiation.cc:407] Connection header received
0924 11:38:47.957834 (+ 2445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:47.957843 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:47.958102 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:47.958295 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:38:47.959150 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:47.960667 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:47.963125 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:47.964710 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:47.964808 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:47.964990 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:47.965580 (+ 590us) server_negotiation.cc:299] Negotiation successful
0924 11:38:47.965753 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":309,"threads_started":1}
I20260924 11:38:47.966851 24667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:47.943891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.3:43361 (local address 127.0.0.1:37000)
0924 11:38:47.956520 (+ 12629us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:47.956537 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:38:47.956642 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:38:47.960047 (+ 3405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:38:47.960051 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:38:47.960078 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:38:47.960292 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:47.960302 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:47.962400 (+ 2098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:47.962408 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:47.964376 (+ 1968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:47.964393 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:47.966211 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:47.966216 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:47.966435 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:47.966475 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 11:38:47.966534 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 11:38:47.966646 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12501,"mutex_wait_us":3,"thread_start_us":13736,"threads_started":1}
I20260924 11:38:47.967921 24671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:47.944307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37000 (local address 127.6.212.3:43361)
0924 11:38:47.959540 (+ 15233us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:47.959551 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:47.959598 (+ 47us) server_negotiation.cc:407] Connection header received
0924 11:38:47.959694 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:47.959701 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:47.959873 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:47.960060 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:38:47.960475 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:47.962069 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:47.964606 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:47.966034 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:47.967098 (+ 1064us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:47.967348 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:47.967436 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 11:38:47.967599 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15020,"thread_start_us":15931,"threads_started":1}
I20260924 11:38:47.970017 24454 tablet_service.cc:1467] Processing CreateTablet for tablet e6a2b637256d4ffcb9199bea02454320 (DEFAULT_TABLE table=test-table [id=4976a2abf20c4b5aabb7cd62d1a7df27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:38:47.971647 24454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6a2b637256d4ffcb9199bea02454320. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:38:47.971998 24588 tablet_service.cc:1467] Processing CreateTablet for tablet e6a2b637256d4ffcb9199bea02454320 (DEFAULT_TABLE table=test-table [id=4976a2abf20c4b5aabb7cd62d1a7df27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:38:47.973366 24588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6a2b637256d4ffcb9199bea02454320. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:38:47.974956 24672 tablet_bootstrap.cc:492] T e6a2b637256d4ffcb9199bea02454320 P 19cac7fb427c4a118d0c7e86bf622b00: Bootstrap starting.
I20260924 11:38:47.978013 24672 tablet_bootstrap.cc:654] T e6a2b637256d4ffcb9199bea02454320 P 19cac7fb427c4a118d0c7e86bf622b00: Neither blocks nor log segments found. Creating new log.
I20260924 11:38:47.979295 24672 log.cc:824] T e6a2b637256d4ffcb9199bea02454320 P 19cac7fb427c4a118d0c7e86bf622b00: Log is configured to *not* fsync() on all Append() calls
I20260924 11:38:47.981946 24672 tablet_bootstrap.cc:492] T e6a2b637256d4ffcb9199bea02454320 P 19cac7fb427c4a118d0c7e86bf622b00: No bootstrap required, opened a new log
I20260924 11:38:47.982198 24672 ts_tablet_manager.cc:1397] T e6a2b637256d4ffcb9199bea02454320 P 19cac7fb427c4a118d0c7e86bf622b00: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 11:38:47.986377 24674 tablet_bootstrap.cc:492] T e6a2b637256d4ffcb9199bea02454320 P 56e40517e1654c698471e47ed9fba4f9: Bootstrap starting.
I20260924 11:38:47.988130 24675 tablet_bootstrap.cc:492] T e6a2b637256d4ffcb9199bea02454320 P b775d8ccb23a433ebf09ca00459dc55a: Bootstrap starting.
I20260924 11:38:47.990196 24674 tablet_bootstrap.cc:654] T e6a2b637256d4ffcb9199bea02454320 P 56e40517e1654c698471e47ed9fba4f9: Neither blocks nor log segments found. Creating new log.
I20260924 11:38:47.991092 24675 tablet_bootstrap.cc:654] T e6a2b637256d4ffcb9199bea02454320 P b775d8ccb23a433ebf09ca00459dc55a: Neither blocks nor log segments found. Creating new log.
I20260924 11:38:47.991474 24674 log.cc:824] T e6a2b637256d4ffcb9199bea02454320 P 56e40517e1654c698471e47ed9fba4f9: Log is configured to *not* fsync() on all Append() calls
I20260924 11:38:47.992018 24672 raft_consensus.cc:348] T e6a2b637256d4ffcb9199bea02454320 P 19cac7fb427c4a118d0c7e86bf622b00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19cac7fb427c4a118d0c7e86bf622b00" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 34557 } } peers { permanent_uuid: "56e40517e1654c698471e47ed9fba4f9" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 46589 } } peers { permanent_uuid: "b775d8ccb23a433ebf09ca00459dc55a" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43361 } }
I20260924 11:38:47.992322 24675 log.cc:824] T e6a2b637256d4ffcb9199bea02454320 P b775d8ccb23a433ebf09ca00459dc55a: Log is configured to *not* fsync() on all Append() calls
I20260924 11:38:47.992448 24672 raft_consensus.cc:374] T e6a2b637256d4ffcb9199bea02454320 P 19cac7fb427c4a118d0c7e86bf622b00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:38:47.992580 24672 raft_consensus.cc:729] T e6a2b637256d4ffcb9199bea02454320 P 19cac7fb427c4a118d0c7e86bf622b00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19cac7fb427c4a118d0c7e86bf622b00, State: Initialized, Role: FOLLOWER
I20260924 11:38:47.993414 24672 consensus_queue.cc:260] T e6a2b637256d4ffcb9199bea02454320 P 19cac7fb427c4a118d0c7e86bf622b00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19cac7fb427c4a118d0c7e86bf622b00" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 34557 } } peers { permanent_uuid: "56e40517e1654c698471e47ed9fba4f9" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 46589 } } peers { permanent_uuid: "b775d8ccb23a433ebf09ca00459dc55a" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43361 } }
I20260924 11:38:47.995105 24675 tablet_bootstrap.cc:492] T e6a2b637256d4ffcb9199bea02454320 P b775d8ccb23a433ebf09ca00459dc55a: No bootstrap required, opened a new log
I20260924 11:38:47.995417 24675 ts_tablet_manager.cc:1397] T e6a2b637256d4ffcb9199bea02454320 P b775d8ccb23a433ebf09ca00459dc55a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 11:38:47.996966 24672 ts_tablet_manager.cc:1428] T e6a2b637256d4ffcb9199bea02454320 P 19cac7fb427c4a118d0c7e86bf622b00: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260924 11:38:47.999918 24674 tablet_bootstrap.cc:492] T e6a2b637256d4ffcb9199bea02454320 P 56e40517e1654c698471e47ed9fba4f9: No bootstrap required, opened a new log
I20260924 11:38:48.000196 24674 ts_tablet_manager.cc:1397] T e6a2b637256d4ffcb9199bea02454320 P 56e40517e1654c698471e47ed9fba4f9: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260924 11:38:48.004010 24675 raft_consensus.cc:348] T e6a2b637256d4ffcb9199bea02454320 P b775d8ccb23a433ebf09ca00459dc55a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19cac7fb427c4a118d0c7e86bf622b00" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 34557 } } peers { permanent_uuid: "56e40517e1654c698471e47ed9fba4f9" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 46589 } } peers { permanent_uuid: "b775d8ccb23a433ebf09ca00459dc55a" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43361 } }
I20260924 11:38:48.004424 24675 raft_consensus.cc:374] T e6a2b637256d4ffcb9199bea02454320 P b775d8ccb23a433ebf09ca00459dc55a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:38:48.004534 24675 raft_consensus.cc:729] T e6a2b637256d4ffcb9199bea02454320 P b775d8ccb23a433ebf09ca00459dc55a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b775d8ccb23a433ebf09ca00459dc55a, State: Initialized, Role: FOLLOWER
I20260924 11:38:48.005090 24675 consensus_queue.cc:260] T e6a2b637256d4ffcb9199bea02454320 P b775d8ccb23a433ebf09ca00459dc55a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19cac7fb427c4a118d0c7e86bf622b00" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 34557 } } peers { permanent_uuid: "56e40517e1654c698471e47ed9fba4f9" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 46589 } } peers { permanent_uuid: "b775d8ccb23a433ebf09ca00459dc55a" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43361 } }
I20260924 11:38:48.006147 24653 heartbeater.cc:503] Master 127.6.212.62:46353 was elected leader, sending a full tablet report...
I20260924 11:38:48.006754 24675 ts_tablet_manager.cc:1428] T e6a2b637256d4ffcb9199bea02454320 P b775d8ccb23a433ebf09ca00459dc55a: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20260924 11:38:48.010093 24674 raft_consensus.cc:348] T e6a2b637256d4ffcb9199bea02454320 P 56e40517e1654c698471e47ed9fba4f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19cac7fb427c4a118d0c7e86bf622b00" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 34557 } } peers { permanent_uuid: "56e40517e1654c698471e47ed9fba4f9" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 46589 } } peers { permanent_uuid: "b775d8ccb23a433ebf09ca00459dc55a" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43361 } }
I20260924 11:38:48.010598 24674 raft_consensus.cc:374] T e6a2b637256d4ffcb9199bea02454320 P 56e40517e1654c698471e47ed9fba4f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:38:48.010713 24674 raft_consensus.cc:729] T e6a2b637256d4ffcb9199bea02454320 P 56e40517e1654c698471e47ed9fba4f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e40517e1654c698471e47ed9fba4f9, State: Initialized, Role: FOLLOWER
I20260924 11:38:48.011659 24674 consensus_queue.cc:260] T e6a2b637256d4ffcb9199bea02454320 P 56e40517e1654c698471e47ed9fba4f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19cac7fb427c4a118d0c7e86bf622b00" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 34557 } } peers { permanent_uuid: "56e40517e1654c698471e47ed9fba4f9" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 46589 } } peers { permanent_uuid: "b775d8ccb23a433ebf09ca00459dc55a" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43361 } }
I20260924 11:38:48.012856 24519 heartbeater.cc:503] Master 127.6.212.62:46353 was elected leader, sending a full tablet report...
I20260924 11:38:48.013418 24674 ts_tablet_manager.cc:1428] T e6a2b637256d4ffcb9199bea02454320 P 56e40517e1654c698471e47ed9fba4f9: Time spent starting tablet: real 0.013s user 0.008s sys 0.003s
W20260924 11:38:48.043246 24386 tablet.cc:2369] T e6a2b637256d4ffcb9199bea02454320 P 19cac7fb427c4a118d0c7e86bf622b00: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:38:48.082966 24678 raft_consensus.cc:482] T e6a2b637256d4ffcb9199bea02454320 P 19cac7fb427c4a118d0c7e86bf622b00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:38:48.083496 24678 raft_consensus.cc:504] T e6a2b637256d4ffcb9199bea02454320 P 19cac7fb427c4a118d0c7e86bf622b00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19cac7fb427c4a118d0c7e86bf622b00" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 34557 } } peers { permanent_uuid: "56e40517e1654c698471e47ed9fba4f9" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 46589 } } peers { permanent_uuid: "b775d8ccb23a433ebf09ca00459dc55a" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43361 } }
I20260924 11:38:48.085446 24678 leader_election.cc:288] T e6a2b637256d4ffcb9199bea02454320 P 19cac7fb427c4a118d0c7e86bf622b00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56e40517e1654c698471e47ed9fba4f9 (127.6.212.2:46589), b775d8ccb23a433ebf09ca00459dc55a (127.6.212.3:43361)
I20260924 11:38:48.087642 24680 raft_consensus.cc:482] T e6a2b637256d4ffcb9199bea02454320 P 56e40517e1654c698471e47ed9fba4f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:38:48.087950 24680 raft_consensus.cc:504] T e6a2b637256d4ffcb9199bea02454320 P 56e40517e1654c698471e47ed9fba4f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19cac7fb427c4a118d0c7e86bf622b00" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 34557 } } peers { permanent_uuid: "56e40517e1654c698471e47ed9fba4f9" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 46589 } } peers { permanent_uuid: "b775d8ccb23a433ebf09ca00459dc55a" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43361 } }
W20260924 11:38:48.088256 24520 tablet.cc:2369] T e6a2b637256d4ffcb9199bea02454320 P 56e40517e1654c698471e47ed9fba4f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:38:48.096050 24680 leader_election.cc:288] T e6a2b637256d4ffcb9199bea02454320 P 56e40517e1654c698471e47ed9fba4f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19cac7fb427c4a118d0c7e86bf622b00 (127.6.212.1:34557), b775d8ccb23a433ebf09ca00459dc55a (127.6.212.3:43361)
I20260924 11:38:48.096655 24669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:48.085943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:47551 (local address 127.6.212.2:46589)
0924 11:38:48.086137 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:48.086141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:48.086731 (+ 590us) server_negotiation.cc:407] Connection header received
0924 11:38:48.086979 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:48.086986 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:48.087089 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:48.087196 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:38:48.088759 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:48.089852 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:48.093078 (+ 3226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:48.094933 (+ 1855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:48.095093 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:48.095233 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:48.096041 (+ 808us) server_negotiation.cc:299] Negotiation successful
0924 11:38:48.096236 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 11:38:48.099416 24681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:48.085898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.2:46589 (local address 127.6.212.1:47551)
0924 11:38:48.086599 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:48.086619 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 11:38:48.086760 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:38:48.087431 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:38:48.087436 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:38:48.087493 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:38:48.088608 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:48.088617 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:48.090018 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:48.090024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:48.092840 (+ 2816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:48.092862 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:48.095502 (+ 2640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:48.095511 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:48.095790 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:48.095845 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 11:38:48.096856 (+ 1011us) client_negotiation.cc:240] Negotiation successful
0924 11:38:48.098914 (+ 2058us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"mutex_wait_us":1859,"thread_start_us":387,"threads_started":1}
I20260924 11:38:48.100519 24474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a2b637256d4ffcb9199bea02454320" candidate_uuid: "19cac7fb427c4a118d0c7e86bf622b00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e40517e1654c698471e47ed9fba4f9" is_pre_election: true
I20260924 11:38:48.101174 24474 raft_consensus.cc:2454] T e6a2b637256d4ffcb9199bea02454320 P 56e40517e1654c698471e47ed9fba4f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19cac7fb427c4a118d0c7e86bf622b00 in term 0.
I20260924 11:38:48.101739 24671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:48.087114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:33727 (local address 127.6.212.3:43361)
0924 11:38:48.087313 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:48.087322 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:48.087599 (+ 277us) server_negotiation.cc:407] Connection header received
0924 11:38:48.087812 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:48.087832 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:48.087915 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:48.088012 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:38:48.088588 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:48.089816 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:48.098845 (+ 9029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:48.100289 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:48.100439 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:48.100565 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:48.101330 (+ 765us) server_negotiation.cc:299] Negotiation successful
0924 11:38:48.101481 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":75}
I20260924 11:38:48.101950 24274 leader_election.cc:302] T e6a2b637256d4ffcb9199bea02454320 P 19cac7fb427c4a118d0c7e86bf622b00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19cac7fb427c4a118d0c7e86bf622b00, 56e40517e1654c698471e47ed9fba4f9; no voters:
I20260924 11:38:48.102118 24682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:48.086735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.3:43361 (local address 127.6.212.1:33727)
0924 11:38:48.087478 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:48.087498 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 11:38:48.087662 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:38:48.088026 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:38:48.088030 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:38:48.088052 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:38:48.088430 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:48.088447 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:48.090018 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:48.090024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:48.098641 (+ 8617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:48.098659 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:48.100799 (+ 2140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:48.100808 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:48.101106 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:48.101144 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 11:38:48.101294 (+ 150us) client_negotiation.cc:240] Negotiation successful
0924 11:38:48.101525 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"mutex_wait_us":6164,"thread_start_us":353,"threads_started":1}
I20260924 11:38:48.102476 24678 raft_consensus.cc:2781] T e6a2b637256d4ffcb9199bea02454320 P 19cac7fb427c4a118d0c7e86bf622b00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:38:48.102629 24678 raft_consensus.cc:482] T e6a2b637256d4ffcb9199bea02454320 P 19cac7fb427c4a118d0c7e86bf622b00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:38:48.102741 24678 raft_consensus.cc:3037] T e6a2b637256d4ffcb9199bea02454320 P 19cac7fb427c4a118d0c7e86bf622b00 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:38:48.103498 24608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a2b637256d4ffcb9199bea02454320" candidate_uuid: "19cac7fb427c4a118d0c7e86bf622b00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b775d8ccb23a433ebf09ca00459dc55a" is_pre_election: true
I20260924 11:38:48.104106 24608 raft_consensus.cc:2454] T e6a2b637256d4ffcb9199bea02454320 P b775d8ccb23a433ebf09ca00459dc55a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19cac7fb427c4a118d0c7e86bf622b00 in term 0.
I20260924 11:38:48.105854 24668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:48.090743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:43967 (local address 127.6.212.1:34557)
0924 11:38:48.091013 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:48.091021 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:48.097533 (+ 6512us) server_negotiation.cc:407] Connection header received
0924 11:38:48.097843 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:48.097850 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:48.097936 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:48.098024 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:38:48.098674 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:48.099920 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:48.103398 (+ 3478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:48.104738 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:48.104843 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:48.104933 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:48.105511 (+ 578us) server_negotiation.cc:299] Negotiation successful
0924 11:38:48.105631 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 11:38:48.105885 24683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:48.090297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.3:43361 (local address 127.6.212.2:35055)
0924 11:38:48.097843 (+ 7546us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:48.097860 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:38:48.097966 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:38:48.098503 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:38:48.098507 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:38:48.098528 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:38:48.098905 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:48.098916 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:48.100791 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:48.100796 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:48.102987 (+ 2191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:48.103000 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:48.105077 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:48.105083 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:48.105270 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:48.105303 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 11:38:48.105394 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 11:38:48.105548 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7417,"thread_start_us":6745,"threads_started":1}
I20260924 11:38:48.106096 24678 raft_consensus.cc:504] T e6a2b637256d4ffcb9199bea02454320 P 19cac7fb427c4a118d0c7e86bf622b00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19cac7fb427c4a118d0c7e86bf622b00" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 34557 } } peers { permanent_uuid: "56e40517e1654c698471e47ed9fba4f9" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 46589 } } peers { permanent_uuid: "b775d8ccb23a433ebf09ca00459dc55a" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43361 } }
I20260924 11:38:48.106359 24685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:48.090837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:35055 (local address 127.6.212.3:43361)
0924 11:38:48.098199 (+ 7362us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:48.098205 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:48.098229 (+ 24us) server_negotiation.cc:407] Connection header received
0924 11:38:48.098290 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:48.098295 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:48.098359 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:48.098465 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:38:48.099105 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:48.100625 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:48.103172 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:48.104691 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:48.104819 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:48.104919 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:48.105438 (+ 519us) server_negotiation.cc:299] Negotiation successful
0924 11:38:48.105579 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":7211,"thread_start_us":7782,"threads_started":1}
I20260924 11:38:48.106586 24684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:48.090174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:34557 (local address 127.6.212.2:43967)
0924 11:38:48.097427 (+ 7253us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:48.097448 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 11:38:48.097653 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:38:48.098176 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:38:48.098179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:38:48.098203 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:38:48.098500 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:48.098509 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:48.100164 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:48.100172 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:48.103175 (+ 3003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:48.103186 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:48.105196 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:48.105203 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:48.105343 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:48.105384 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 11:38:48.106073 (+ 689us) client_negotiation.cc:240] Negotiation successful
0924 11:38:48.106208 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7097,"mutex_wait_us":710,"thread_start_us":7320,"threads_started":1}
I20260924 11:38:48.106797 24678 leader_election.cc:288] T e6a2b637256d4ffcb9199bea02454320 P 19cac7fb427c4a118d0c7e86bf622b00 [CANDIDATE]: Term 1 election: Requested vote from peers 56e40517e1654c698471e47ed9fba4f9 (127.6.212.2:46589), b775d8ccb23a433ebf09ca00459dc55a (127.6.212.3:43361)
I20260924 11:38:48.107266 24608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a2b637256d4ffcb9199bea02454320" candidate_uuid: "19cac7fb427c4a118d0c7e86bf622b00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b775d8ccb23a433ebf09ca00459dc55a"
I20260924 11:38:48.107508 24608 raft_consensus.cc:3037] T e6a2b637256d4ffcb9199bea02454320 P b775d8ccb23a433ebf09ca00459dc55a [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:38:48.107553 24607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a2b637256d4ffcb9199bea02454320" candidate_uuid: "56e40517e1654c698471e47ed9fba4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b775d8ccb23a433ebf09ca00459dc55a" is_pre_election: true
I20260924 11:38:48.107784 24474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a2b637256d4ffcb9199bea02454320" candidate_uuid: "19cac7fb427c4a118d0c7e86bf622b00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e40517e1654c698471e47ed9fba4f9"
I20260924 11:38:48.107985 24474 raft_consensus.cc:3037] T e6a2b637256d4ffcb9199bea02454320 P 56e40517e1654c698471e47ed9fba4f9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:38:48.108583 24340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a2b637256d4ffcb9199bea02454320" candidate_uuid: "56e40517e1654c698471e47ed9fba4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19cac7fb427c4a118d0c7e86bf622b00" is_pre_election: true
I20260924 11:38:48.109014 24340 raft_consensus.cc:2379] T e6a2b637256d4ffcb9199bea02454320 P 19cac7fb427c4a118d0c7e86bf622b00 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 56e40517e1654c698471e47ed9fba4f9 in current term 1: Already voted for candidate 19cac7fb427c4a118d0c7e86bf622b00 in this term.
I20260924 11:38:48.109784 24410 leader_election.cc:302] T e6a2b637256d4ffcb9199bea02454320 P 56e40517e1654c698471e47ed9fba4f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 56e40517e1654c698471e47ed9fba4f9; no voters: 19cac7fb427c4a118d0c7e86bf622b00, b775d8ccb23a433ebf09ca00459dc55a
I20260924 11:38:48.110675 24608 raft_consensus.cc:2454] T e6a2b637256d4ffcb9199bea02454320 P b775d8ccb23a433ebf09ca00459dc55a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19cac7fb427c4a118d0c7e86bf622b00 in term 1.
I20260924 11:38:48.111011 24274 leader_election.cc:302] T e6a2b637256d4ffcb9199bea02454320 P 19cac7fb427c4a118d0c7e86bf622b00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19cac7fb427c4a118d0c7e86bf622b00, b775d8ccb23a433ebf09ca00459dc55a; no voters:
I20260924 11:38:48.111088 24474 raft_consensus.cc:2454] T e6a2b637256d4ffcb9199bea02454320 P 56e40517e1654c698471e47ed9fba4f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19cac7fb427c4a118d0c7e86bf622b00 in term 1.
I20260924 11:38:48.111253 24680 raft_consensus.cc:2726] T e6a2b637256d4ffcb9199bea02454320 P 56e40517e1654c698471e47ed9fba4f9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 11:38:48.111277 24678 raft_consensus.cc:2781] T e6a2b637256d4ffcb9199bea02454320 P 19cac7fb427c4a118d0c7e86bf622b00 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:38:48.114709 24678 raft_consensus.cc:686] T e6a2b637256d4ffcb9199bea02454320 P 19cac7fb427c4a118d0c7e86bf622b00 [term 1 LEADER]: Becoming Leader. State: Replica: 19cac7fb427c4a118d0c7e86bf622b00, State: Running, Role: LEADER
I20260924 11:38:48.115190 24678 consensus_queue.cc:237] T e6a2b637256d4ffcb9199bea02454320 P 19cac7fb427c4a118d0c7e86bf622b00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19cac7fb427c4a118d0c7e86bf622b00" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 34557 } } peers { permanent_uuid: "56e40517e1654c698471e47ed9fba4f9" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 46589 } } peers { permanent_uuid: "b775d8ccb23a433ebf09ca00459dc55a" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43361 } }
W20260924 11:38:48.116752 24654 tablet.cc:2369] T e6a2b637256d4ffcb9199bea02454320 P b775d8ccb23a433ebf09ca00459dc55a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:38:48.119681 24195 catalog_manager.cc:5495] T e6a2b637256d4ffcb9199bea02454320 P 19cac7fb427c4a118d0c7e86bf622b00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19cac7fb427c4a118d0c7e86bf622b00 (127.6.212.1). New cstate: current_term: 1 leader_uuid: "19cac7fb427c4a118d0c7e86bf622b00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19cac7fb427c4a118d0c7e86bf622b00" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 34557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56e40517e1654c698471e47ed9fba4f9" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 46589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b775d8ccb23a433ebf09ca00459dc55a" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43361 } health_report { overall_health: UNKNOWN } } }
I20260924 11:38:48.164958 24668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:48.159422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50716 (local address 127.6.212.1:34557)
0924 11:38:48.159633 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:48.159639 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:48.159754 (+ 115us) server_negotiation.cc:407] Connection header received
0924 11:38:48.159996 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:48.160001 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:48.160084 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:48.160233 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:38:48.160615 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:48.161372 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:48.162777 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:48.163197 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:48.163327 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:48.163867 (+ 540us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:38:48.164363 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:38:48.164517 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:48.164584 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 11:38:48.164665 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 11:38:48.172960 24474 raft_consensus.cc:1261] T e6a2b637256d4ffcb9199bea02454320 P 56e40517e1654c698471e47ed9fba4f9 [term 1 FOLLOWER]: Refusing update from remote peer 19cac7fb427c4a118d0c7e86bf622b00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:38:48.173106 24608 raft_consensus.cc:1261] T e6a2b637256d4ffcb9199bea02454320 P b775d8ccb23a433ebf09ca00459dc55a [term 1 FOLLOWER]: Refusing update from remote peer 19cac7fb427c4a118d0c7e86bf622b00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:38:48.173904 24686 consensus_queue.cc:1035] T e6a2b637256d4ffcb9199bea02454320 P 19cac7fb427c4a118d0c7e86bf622b00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e40517e1654c698471e47ed9fba4f9" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 46589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:38:48.174254 24678 consensus_queue.cc:1035] T e6a2b637256d4ffcb9199bea02454320 P 19cac7fb427c4a118d0c7e86bf622b00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b775d8ccb23a433ebf09ca00459dc55a" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:38:48.190683 24691 mvcc.cc:205] Tried to move back new op lower bound from 7332863705786593280 to 7332863705563705344. Current Snapshot: MvccSnapshot[applied={T|T < 7332863705786593280}]
I20260924 11:38:48.200286 24194 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:38:48.200580 24194 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:38:48.204602 24194 catalog_manager.cc:5782] T 00000000000000000000000000000000 P df84b80e0fc74ce692c6f00e60e2886a: Sending DeleteTablet for 3 replicas of tablet e6a2b637256d4ffcb9199bea02454320
I20260924 11:38:48.205880 24320 tablet_service.cc:1514] Processing DeleteTablet for tablet e6a2b637256d4ffcb9199bea02454320 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:38:48 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:50706
I20260924 11:38:48.205991 24588 tablet_service.cc:1514] Processing DeleteTablet for tablet e6a2b637256d4ffcb9199bea02454320 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:38:48 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:37000
I20260924 11:38:48.205991 24454 tablet_service.cc:1514] Processing DeleteTablet for tablet e6a2b637256d4ffcb9199bea02454320 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:38:48 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:56300
I20260924 11:38:48.208165 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 24257
I20260924 11:38:48.219962 24698 tablet_replica.cc:330] T e6a2b637256d4ffcb9199bea02454320 P 19cac7fb427c4a118d0c7e86bf622b00: stopping tablet replica
I20260924 11:38:48.220590 24698 raft_consensus.cc:2229] T e6a2b637256d4ffcb9199bea02454320 P 19cac7fb427c4a118d0c7e86bf622b00 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:38:48.221450 24698 raft_consensus.cc:2258] T e6a2b637256d4ffcb9199bea02454320 P 19cac7fb427c4a118d0c7e86bf622b00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:38:48.222762 24700 tablet_replica.cc:330] T e6a2b637256d4ffcb9199bea02454320 P b775d8ccb23a433ebf09ca00459dc55a: stopping tablet replica
I20260924 11:38:48.223141 24699 tablet_replica.cc:330] T e6a2b637256d4ffcb9199bea02454320 P 56e40517e1654c698471e47ed9fba4f9: stopping tablet replica
I20260924 11:38:48.223320 24700 raft_consensus.cc:2229] T e6a2b637256d4ffcb9199bea02454320 P b775d8ccb23a433ebf09ca00459dc55a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:38:48.223672 24699 raft_consensus.cc:2229] T e6a2b637256d4ffcb9199bea02454320 P 56e40517e1654c698471e47ed9fba4f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:38:48.223968 24700 raft_consensus.cc:2258] T e6a2b637256d4ffcb9199bea02454320 P b775d8ccb23a433ebf09ca00459dc55a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:38:48.224257 24699 raft_consensus.cc:2258] T e6a2b637256d4ffcb9199bea02454320 P 56e40517e1654c698471e47ed9fba4f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:38:48.224951 24698 ts_tablet_manager.cc:1905] T e6a2b637256d4ffcb9199bea02454320 P 19cac7fb427c4a118d0c7e86bf622b00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:38:48.226487 24700 ts_tablet_manager.cc:1905] T e6a2b637256d4ffcb9199bea02454320 P b775d8ccb23a433ebf09ca00459dc55a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:38:48.229007 24699 ts_tablet_manager.cc:1905] T e6a2b637256d4ffcb9199bea02454320 P 56e40517e1654c698471e47ed9fba4f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:38:48.233294 24698 ts_tablet_manager.cc:1918] T e6a2b637256d4ffcb9199bea02454320 P 19cac7fb427c4a118d0c7e86bf622b00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:38:48.233486 24698 log.cc:1196] T e6a2b637256d4ffcb9199bea02454320 P 19cac7fb427c4a118d0c7e86bf622b00: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-0/wal/wals/e6a2b637256d4ffcb9199bea02454320
I20260924 11:38:48.234081 24698 ts_tablet_manager.cc:1939] T e6a2b637256d4ffcb9199bea02454320 P 19cac7fb427c4a118d0c7e86bf622b00: Deleting consensus metadata
Sep 24 11:38:48 dist-test-slave-cq7c krb5kdc[24151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.1@KRBTEST.COM
I20260924 11:38:48.236060 24183 catalog_manager.cc:4841] TS 19cac7fb427c4a118d0c7e86bf622b00 (127.6.212.1:34557): tablet e6a2b637256d4ffcb9199bea02454320 (table test-table [id=4976a2abf20c4b5aabb7cd62d1a7df27]) successfully deleted
I20260924 11:38:48.239279 24700 ts_tablet_manager.cc:1918] T e6a2b637256d4ffcb9199bea02454320 P b775d8ccb23a433ebf09ca00459dc55a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:38:48.239497 24700 log.cc:1196] T e6a2b637256d4ffcb9199bea02454320 P b775d8ccb23a433ebf09ca00459dc55a: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-2/wal/wals/e6a2b637256d4ffcb9199bea02454320
I20260924 11:38:48.240026 24700 ts_tablet_manager.cc:1939] T e6a2b637256d4ffcb9199bea02454320 P b775d8ccb23a433ebf09ca00459dc55a: Deleting consensus metadata
I20260924 11:38:48.240695 24699 ts_tablet_manager.cc:1918] T e6a2b637256d4ffcb9199bea02454320 P 56e40517e1654c698471e47ed9fba4f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:38:48.240854 24699 log.cc:1196] T e6a2b637256d4ffcb9199bea02454320 P 56e40517e1654c698471e47ed9fba4f9: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790249617057452-6992-0/minicluster-data/ts-1/wal/wals/e6a2b637256d4ffcb9199bea02454320
I20260924 11:38:48.241425 24699 ts_tablet_manager.cc:1939] T e6a2b637256d4ffcb9199bea02454320 P 56e40517e1654c698471e47ed9fba4f9: Deleting consensus metadata
I20260924 11:38:48.241942 24181 catalog_manager.cc:4841] TS b775d8ccb23a433ebf09ca00459dc55a (127.6.212.3:43361): tablet e6a2b637256d4ffcb9199bea02454320 (table test-table [id=4976a2abf20c4b5aabb7cd62d1a7df27]) successfully deleted
I20260924 11:38:48.243234 24181 catalog_manager.cc:4841] TS 56e40517e1654c698471e47ed9fba4f9 (127.6.212.2:46589): tablet e6a2b637256d4ffcb9199bea02454320 (table test-table [id=4976a2abf20c4b5aabb7cd62d1a7df27]) successfully deleted
I20260924 11:38:48.245209 24668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:48.209249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50720 (local address 127.6.212.1:34557)
0924 11:38:48.209431 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:48.209437 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:48.222368 (+ 12931us) server_negotiation.cc:407] Connection header received
0924 11:38:48.222682 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:48.222687 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:48.222783 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:48.222871 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:38:48.226229 (+ 3358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:48.227470 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:48.229922 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:48.230646 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:48.230777 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:48.236787 (+ 6010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:38:48.236800 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:38:48.236818 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:38:48.236869 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:38:48.241100 (+ 4231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:48.242620 (+ 1520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:48.242633 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:48.242638 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:48.242762 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:48.243246 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:48.243252 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:48.243256 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:48.243813 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:38:48.244128 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:48.244543 (+ 415us) server_negotiation.cc:299] Negotiation successful
0924 11:38:48.244813 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 11:38:48.246138 24380 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:38:48.350009 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 24257
I20260924 11:38:48.370841 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 24391
Sep 24 11:38:48 dist-test-slave-cq7c krb5kdc[24151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.2@KRBTEST.COM
I20260924 11:38:48.390492 24669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:48.371734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56304 (local address 127.6.212.2:46589)
0924 11:38:48.371903 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:48.371908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:48.371927 (+ 19us) server_negotiation.cc:407] Connection header received
0924 11:38:48.372034 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:48.372038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:48.372099 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:48.372180 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:38:48.373663 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:48.374383 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:48.375815 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:48.376236 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:48.376327 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:48.384565 (+ 8238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:38:48.384580 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:38:48.384596 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:38:48.384634 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:38:48.387814 (+ 3180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:48.388567 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:48.388574 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:48.388577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:48.388707 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:48.389117 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:48.389120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:48.389122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:48.389482 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:38:48.389692 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:48.389994 (+ 302us) server_negotiation.cc:299] Negotiation successful
0924 11:38:48.390152 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 11:38:48.391150 24514 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:38:48.483068 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 24391
I20260924 11:38:48.502377 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 24525
Sep 24 11:38:48 dist-test-slave-cq7c krb5kdc[24151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.3@KRBTEST.COM
I20260924 11:38:48.517817 24685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:48.503181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37010 (local address 127.6.212.3:43361)
0924 11:38:48.503397 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:48.503404 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:48.503474 (+ 70us) server_negotiation.cc:407] Connection header received
0924 11:38:48.503542 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:48.503546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:48.503613 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:48.503704 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:38:48.505085 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:48.505838 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:48.507078 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:48.507746 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:48.507850 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:48.512198 (+ 4348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:38:48.512214 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:38:48.512230 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:38:48.512269 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:38:48.515215 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:48.515904 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:48.515911 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:48.515914 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:48.516055 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:48.516375 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:48.516378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:48.516380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:48.516731 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:38:48.516975 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:48.517280 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 11:38:48.517519 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 11:38:48.518484 24648 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:38:48.611032 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 24525
I20260924 11:38:48.627691 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 24164
I20260924 11:38:48.628799 24226 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:38:48.742142 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 24164
2026-09-24T11:38:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4481 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T11:38:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:38:48Z Disabled control of system clock
I20260924 11:38:48.823632 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:43633
--webserver_interface=127.6.212.62
--webserver_port=0
--builtin_ntp_servers=127.6.212.20:40757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:43633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 11:38:49.173084 24709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:38:49.173393 24709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:38:49.173533 24709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:38:49.181159 24709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:38:49.181281 24709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:38:49.181319 24709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:38:49.181429 24709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:38:49.181495 24709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:38:49.191399 24709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:40757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:43633
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:43633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.212.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:38:49.192760 24709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:38:49.194660 24709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:38:49.206846 24714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:38:49.206959 24715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:38:49.206982 24717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:38:49.207799 24709 server_base.cc:1034] running on GCE node
I20260924 11:38:49.208665 24709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:38:49.210270 24709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:38:49.211483 24709 hybrid_clock.cc:648] HybridClock initialized: now 1790249929211430 us; error 47 us; skew 500 ppm
I20260924 11:38:49.211825 24709 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:38:49.214395 24709 webserver.cc:460] Webserver started at http://127.6.212.62:46737/ using document root <none> and password file <none>
I20260924 11:38:49.215181 24709 fs_manager.cc:359] Metadata directory not provided
I20260924 11:38:49.215273 24709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:38:49.215616 24709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:38:49.218688 24709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "9cb443dbf3f44831b10bf7f86705559c"
format_stamp: "Formatted at 2026-09-24 11:38:49 on dist-test-slave-cq7c"
server_key: "0d2c771c7b3d07eb9ad006966e5af9dc"
server_key_iv: "cbb30e196aa5490d61ccc5d64bf4aa55"
server_key_version: "encryptionkey@0"
I20260924 11:38:49.219527 24709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "9cb443dbf3f44831b10bf7f86705559c"
format_stamp: "Formatted at 2026-09-24 11:38:49 on dist-test-slave-cq7c"
server_key: "0d2c771c7b3d07eb9ad006966e5af9dc"
server_key_iv: "cbb30e196aa5490d61ccc5d64bf4aa55"
server_key_version: "encryptionkey@0"
I20260924 11:38:49.225147 24709 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 11:38:49.229132 24723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:49.230832 24709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 11:38:49.231150 24709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "9cb443dbf3f44831b10bf7f86705559c"
format_stamp: "Formatted at 2026-09-24 11:38:49 on dist-test-slave-cq7c"
server_key: "0d2c771c7b3d07eb9ad006966e5af9dc"
server_key_iv: "cbb30e196aa5490d61ccc5d64bf4aa55"
server_key_version: "encryptionkey@0"
I20260924 11:38:49.231434 24709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:38:49.268817 24709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:38:49.269816 24709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:38:49.270081 24709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:38:49.289324 24709 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.62:43633
I20260924 11:38:49.289357 24774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.62:43633 every 8 connection(s)
I20260924 11:38:49.290735 24709 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:38:49.294763 24775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:38:49.299921 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 24709
I20260924 11:38:49.300117 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:38:49.300645 6992 external_mini_cluster.cc:1428] Setting key 27065d3651172dc1b0fa2cbc4470d3f6
I20260924 11:38:49.306681 24775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cb443dbf3f44831b10bf7f86705559c: Bootstrap starting.
I20260924 11:38:49.310667 24775 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9cb443dbf3f44831b10bf7f86705559c: Neither blocks nor log segments found. Creating new log.
I20260924 11:38:49.311728 24775 log.cc:824] T 00000000000000000000000000000000 P 9cb443dbf3f44831b10bf7f86705559c: Log is configured to *not* fsync() on all Append() calls
I20260924 11:38:49.315421 24775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cb443dbf3f44831b10bf7f86705559c: No bootstrap required, opened a new log
I20260924 11:38:49.317458 24778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:49.302739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33516 (local address 127.6.212.62:43633)
0924 11:38:49.303966 (+ 1227us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:49.303986 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:49.304040 (+ 54us) server_negotiation.cc:407] Connection header received
0924 11:38:49.306430 (+ 2390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:49.306470 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:49.307066 (+ 596us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:49.307536 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:38:49.308045 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:49.310083 (+ 2038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:49.311823 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:49.312838 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:49.312969 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:49.314529 (+ 1560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:38:49.314539 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:38:49.314557 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:38:49.315284 (+ 727us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:38:49.315419 (+ 135us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:38:49.315449 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:38:49.315581 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:49.315831 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 11:38:49.316188 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":384,"threads_started":1}
I20260924 11:38:49.324384 24775 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cb443dbf3f44831b10bf7f86705559c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cb443dbf3f44831b10bf7f86705559c" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43633 } }
I20260924 11:38:49.324744 24775 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9cb443dbf3f44831b10bf7f86705559c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:38:49.324811 24775 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cb443dbf3f44831b10bf7f86705559c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cb443dbf3f44831b10bf7f86705559c, State: Initialized, Role: FOLLOWER
I20260924 11:38:49.325368 24775 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cb443dbf3f44831b10bf7f86705559c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cb443dbf3f44831b10bf7f86705559c" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43633 } }
I20260924 11:38:49.325577 24775 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9cb443dbf3f44831b10bf7f86705559c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:38:49.325670 24775 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cb443dbf3f44831b10bf7f86705559c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:38:49.325795 24775 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cb443dbf3f44831b10bf7f86705559c [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:38:49.328362 24775 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cb443dbf3f44831b10bf7f86705559c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cb443dbf3f44831b10bf7f86705559c" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43633 } }
I20260924 11:38:49.328814 24775 leader_election.cc:302] T 00000000000000000000000000000000 P 9cb443dbf3f44831b10bf7f86705559c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cb443dbf3f44831b10bf7f86705559c; no voters:
I20260924 11:38:49.329478 24775 leader_election.cc:288] T 00000000000000000000000000000000 P 9cb443dbf3f44831b10bf7f86705559c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:38:49.329629 24780 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cb443dbf3f44831b10bf7f86705559c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:38:49.330458 24780 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cb443dbf3f44831b10bf7f86705559c [term 1 LEADER]: Becoming Leader. State: Replica: 9cb443dbf3f44831b10bf7f86705559c, State: Running, Role: LEADER
I20260924 11:38:49.331187 24780 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cb443dbf3f44831b10bf7f86705559c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cb443dbf3f44831b10bf7f86705559c" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43633 } }
I20260924 11:38:49.334705 24781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cb443dbf3f44831b10bf7f86705559c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cb443dbf3f44831b10bf7f86705559c. Latest consensus state: current_term: 1 leader_uuid: "9cb443dbf3f44831b10bf7f86705559c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cb443dbf3f44831b10bf7f86705559c" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43633 } } }
I20260924 11:38:49.335165 24781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cb443dbf3f44831b10bf7f86705559c [sys.catalog]: This master's current role is: LEADER
I20260924 11:38:49.335619 24775 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cb443dbf3f44831b10bf7f86705559c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:38:49.335755 24782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cb443dbf3f44831b10bf7f86705559c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9cb443dbf3f44831b10bf7f86705559c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cb443dbf3f44831b10bf7f86705559c" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43633 } } }
I20260924 11:38:49.336097 24782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cb443dbf3f44831b10bf7f86705559c [sys.catalog]: This master's current role is: LEADER
I20260924 11:38:49.338642 24784 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:38:49.344411 24784 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:38:49.354311 24784 catalog_manager.cc:1347] Generated new cluster ID: 37d522d4ef7f4fcdac14a94f3c76050b
I20260924 11:38:49.354462 24784 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:38:49.374934 24784 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:38:49.376613 24784 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:38:49.384716 24784 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9cb443dbf3f44831b10bf7f86705559c: Generated new TSK 0
I20260924 11:38:49.385988 24784 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:38:49.392338 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.1:0
--local_ip_for_outbound_sockets=127.6.212.1
--webserver_interface=127.6.212.1
--webserver_port=0
--tserver_master_addrs=127.6.212.62:43633
--builtin_ntp_servers=127.6.212.20:40757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 11:38:49.738395 24799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:38:49.738668 24799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:38:49.738759 24799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:38:49.746285 24799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:38:49.746402 24799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:38:49.746515 24799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.1
I20260924 11:38:49.755787 24799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:40757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.212.1
--webserver_port=0
--tserver_master_addrs=127.6.212.62:43633
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.1
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:38:49.757066 24799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:38:49.758800 24799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:38:49.772085 24807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:38:49.772523 24805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:38:49.772545 24804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:38:49.773105 24799 server_base.cc:1034] running on GCE node
I20260924 11:38:49.773712 24799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:38:49.774741 24799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:38:49.775987 24799 hybrid_clock.cc:648] HybridClock initialized: now 1790249929775934 us; error 46 us; skew 500 ppm
I20260924 11:38:49.776309 24799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:38:49.778662 24799 webserver.cc:460] Webserver started at http://127.6.212.1:39747/ using document root <none> and password file <none>
I20260924 11:38:49.779397 24799 fs_manager.cc:359] Metadata directory not provided
I20260924 11:38:49.779522 24799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:38:49.779853 24799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:38:49.782639 24799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-0/data/instance:
uuid: "0b406939cf3a4be29d4b5a6a8c176b1c"
format_stamp: "Formatted at 2026-09-24 11:38:49 on dist-test-slave-cq7c"
server_key: "d846bf18be1ffbc244a01ca0592152ba"
server_key_iv: "5945897f3afda27b155aadf0148bd274"
server_key_version: "encryptionkey@0"
I20260924 11:38:49.783488 24799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance:
uuid: "0b406939cf3a4be29d4b5a6a8c176b1c"
format_stamp: "Formatted at 2026-09-24 11:38:49 on dist-test-slave-cq7c"
server_key: "d846bf18be1ffbc244a01ca0592152ba"
server_key_iv: "5945897f3afda27b155aadf0148bd274"
server_key_version: "encryptionkey@0"
I20260924 11:38:49.788702 24799 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 11:38:49.792780 24813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:49.794093 24799 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:38:49.794368 24799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-0/wal
uuid: "0b406939cf3a4be29d4b5a6a8c176b1c"
format_stamp: "Formatted at 2026-09-24 11:38:49 on dist-test-slave-cq7c"
server_key: "d846bf18be1ffbc244a01ca0592152ba"
server_key_iv: "5945897f3afda27b155aadf0148bd274"
server_key_version: "encryptionkey@0"
I20260924 11:38:49.794597 24799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:38:49.805912 24799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:38:49.806753 24799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:38:49.807030 24799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:38:49.808144 24799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:38:49.809665 24799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:38:49.809788 24799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:49.809916 24799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:38:49.810001 24799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:49.845353 24799 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.1:43803
I20260924 11:38:49.845489 24925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.1:43803 every 8 connection(s)
I20260924 11:38:49.846728 24799 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
I20260924 11:38:49.849443 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 24799
I20260924 11:38:49.849632 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance
I20260924 11:38:49.850322 6992 external_mini_cluster.cc:1428] Setting key f26c95329435d1e86e8a368a730b7890
I20260924 11:38:49.853636 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.2:0
--local_ip_for_outbound_sockets=127.6.212.2
--webserver_interface=127.6.212.2
--webserver_port=0
--tserver_master_addrs=127.6.212.62:43633
--builtin_ntp_servers=127.6.212.20:40757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 11:38:49.861503 24928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:49.849703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:43483 (local address 127.6.212.62:43633)
0924 11:38:49.850204 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:49.850211 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:49.852026 (+ 1815us) server_negotiation.cc:407] Connection header received
0924 11:38:49.854787 (+ 2761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:49.854792 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:49.854964 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:49.855105 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:38:49.856374 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:49.857085 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:49.858632 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:49.859061 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:49.859192 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:49.860646 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:38:49.860656 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:38:49.860660 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:38:49.860679 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:38:49.860727 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:38:49.860740 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:38:49.860837 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:49.861072 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 11:38:49.861171 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":239,"threads_started":1}
I20260924 11:38:49.862668 24929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:49.850551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:43633 (local address 127.6.212.1:43483)
0924 11:38:49.851755 (+ 1204us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:49.851812 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 11:38:49.854320 (+ 2508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:38:49.855506 (+ 1186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:38:49.855516 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:38:49.855613 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:38:49.856210 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:49.856227 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:49.857226 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:49.857236 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:49.858488 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:49.858500 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:49.859282 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:49.859289 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:49.859600 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:49.860316 (+ 716us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:38:49.860319 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:38:49.860372 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:38:49.860458 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:38:49.860845 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:38:49.860858 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:38:49.860869 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 11:38:49.861046 (+ 177us) client_negotiation.cc:240] Negotiation successful
0924 11:38:49.861462 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":388,"threads_started":1}
I20260924 11:38:49.864426 24926 heartbeater.cc:347] Connected to a master server at 127.6.212.62:43633
I20260924 11:38:49.864756 24926 heartbeater.cc:464] Registering TS with master...
I20260924 11:38:49.865554 24926 heartbeater.cc:511] Master 127.6.212.62:43633 requested a full tablet report, sending...
I20260924 11:38:49.867944 24740 ts_manager.cc:194] Registered new tserver with Master: 0b406939cf3a4be29d4b5a6a8c176b1c (127.6.212.1:43803)
I20260924 11:38:49.870312 24740 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.212.1:43483
W20260924 11:38:50.196938 24930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:38:50.197256 24930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:38:50.197346 24930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:38:50.205111 24930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:38:50.205230 24930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:38:50.205343 24930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.2
I20260924 11:38:50.214609 24930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:40757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.212.2
--webserver_port=0
--tserver_master_addrs=127.6.212.62:43633
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.2
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:38:50.216061 24930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:38:50.217819 24930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:38:50.231294 24935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:38:50.231204 24938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:38:50.231179 24936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:38:50.231715 24930 server_base.cc:1034] running on GCE node
I20260924 11:38:50.232609 24930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:38:50.233651 24930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:38:50.234890 24930 hybrid_clock.cc:648] HybridClock initialized: now 1790249930234805 us; error 74 us; skew 500 ppm
I20260924 11:38:50.235255 24930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:38:50.237895 24930 webserver.cc:460] Webserver started at http://127.6.212.2:46117/ using document root <none> and password file <none>
I20260924 11:38:50.238637 24930 fs_manager.cc:359] Metadata directory not provided
I20260924 11:38:50.238770 24930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:38:50.239132 24930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:38:50.242439 24930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-1/data/instance:
uuid: "b77a13d463944ae3813c978ea242c9c4"
format_stamp: "Formatted at 2026-09-24 11:38:50 on dist-test-slave-cq7c"
server_key: "0693dd569839983102b645de3507c494"
server_key_iv: "acd377c2c5eb028dd6e6dc5c458b7e8b"
server_key_version: "encryptionkey@0"
I20260924 11:38:50.243273 24930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance:
uuid: "b77a13d463944ae3813c978ea242c9c4"
format_stamp: "Formatted at 2026-09-24 11:38:50 on dist-test-slave-cq7c"
server_key: "0693dd569839983102b645de3507c494"
server_key_iv: "acd377c2c5eb028dd6e6dc5c458b7e8b"
server_key_version: "encryptionkey@0"
I20260924 11:38:50.248703 24930 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 11:38:50.252398 24944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:50.253844 24930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:38:50.254050 24930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-1/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-1/wal
uuid: "b77a13d463944ae3813c978ea242c9c4"
format_stamp: "Formatted at 2026-09-24 11:38:50 on dist-test-slave-cq7c"
server_key: "0693dd569839983102b645de3507c494"
server_key_iv: "acd377c2c5eb028dd6e6dc5c458b7e8b"
server_key_version: "encryptionkey@0"
I20260924 11:38:50.254285 24930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:38:50.280260 24930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:38:50.281141 24930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:38:50.281435 24930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:38:50.282541 24930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:38:50.284024 24930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:38:50.284176 24930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:50.284307 24930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:38:50.284394 24930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:50.319298 24930 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.2:39635
I20260924 11:38:50.319410 25056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.2:39635 every 8 connection(s)
I20260924 11:38:50.320663 24930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
I20260924 11:38:50.322961 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 24930
I20260924 11:38:50.323153 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance
I20260924 11:38:50.323766 6992 external_mini_cluster.cc:1428] Setting key 2cb9f77cb213b21b289c6ff41f2deebe
I20260924 11:38:50.326959 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.3:0
--local_ip_for_outbound_sockets=127.6.212.3
--webserver_interface=127.6.212.3
--webserver_port=0
--tserver_master_addrs=127.6.212.62:43633
--builtin_ntp_servers=127.6.212.20:40757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 11:38:50.335433 24928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:50.323437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:53163 (local address 127.6.212.62:43633)
0924 11:38:50.323612 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:50.323616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:50.325447 (+ 1831us) server_negotiation.cc:407] Connection header received
0924 11:38:50.327517 (+ 2070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:50.327524 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:50.327614 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:50.327730 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:38:50.328813 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:50.329987 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:50.331801 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:50.332524 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:50.332679 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:50.334359 (+ 1680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:38:50.334369 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:38:50.334377 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:38:50.334429 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:38:50.334475 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:38:50.334495 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:38:50.334592 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:50.334858 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 11:38:50.334977 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 11:38:50.336841 25059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:50.324080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:43633 (local address 127.6.212.2:53163)
0924 11:38:50.325254 (+ 1174us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:50.325299 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 11:38:50.327205 (+ 1906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:38:50.327917 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:38:50.327925 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:38:50.328050 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:38:50.328648 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:50.328663 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:50.330167 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:50.330171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:50.331662 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:50.331675 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:50.332756 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:50.332762 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:50.333083 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:50.333945 (+ 862us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:38:50.333949 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:38:50.334008 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:38:50.334117 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:38:50.334596 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:38:50.334620 (+ 24us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:38:50.334630 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 11:38:50.334831 (+ 201us) client_negotiation.cc:240] Negotiation successful
0924 11:38:50.335278 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":370,"threads_started":1}
I20260924 11:38:50.338488 25057 heartbeater.cc:347] Connected to a master server at 127.6.212.62:43633
I20260924 11:38:50.338806 25057 heartbeater.cc:464] Registering TS with master...
I20260924 11:38:50.339778 25057 heartbeater.cc:511] Master 127.6.212.62:43633 requested a full tablet report, sending...
I20260924 11:38:50.341356 24740 ts_manager.cc:194] Registered new tserver with Master: b77a13d463944ae3813c978ea242c9c4 (127.6.212.2:39635)
I20260924 11:38:50.342648 24740 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.212.2:53163
W20260924 11:38:50.656071 25060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:38:50.656342 25060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:38:50.656431 25060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:38:50.663606 25060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:38:50.663707 25060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:38:50.663810 25060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.3
I20260924 11:38:50.672777 25060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:40757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.212.3
--webserver_port=0
--tserver_master_addrs=127.6.212.62:43633
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.3
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:38:50.674055 25060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:38:50.675806 25060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:38:50.688424 25065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:38:50.688390 25066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:38:50.688354 25068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:38:50.689245 25060 server_base.cc:1034] running on GCE node
I20260924 11:38:50.689868 25060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:38:50.690891 25060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:38:50.692117 25060 hybrid_clock.cc:648] HybridClock initialized: now 1790249930692060 us; error 50 us; skew 500 ppm
I20260924 11:38:50.692440 25060 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:38:50.694720 25060 webserver.cc:460] Webserver started at http://127.6.212.3:40411/ using document root <none> and password file <none>
I20260924 11:38:50.695482 25060 fs_manager.cc:359] Metadata directory not provided
I20260924 11:38:50.695608 25060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:38:50.695935 25060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:38:50.698887 25060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-2/data/instance:
uuid: "7068db6e067942d0bc7a39536ca0902d"
format_stamp: "Formatted at 2026-09-24 11:38:50 on dist-test-slave-cq7c"
server_key: "703356c79d842a0e7e159ecb76edaab1"
server_key_iv: "5695e6573838a6674ef37c5e217aecbb"
server_key_version: "encryptionkey@0"
I20260924 11:38:50.699780 25060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance:
uuid: "7068db6e067942d0bc7a39536ca0902d"
format_stamp: "Formatted at 2026-09-24 11:38:50 on dist-test-slave-cq7c"
server_key: "703356c79d842a0e7e159ecb76edaab1"
server_key_iv: "5695e6573838a6674ef37c5e217aecbb"
server_key_version: "encryptionkey@0"
I20260924 11:38:50.704906 25060 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260924 11:38:50.708442 25074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:50.709784 25060 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 11:38:50.709954 25060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-2/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-2/wal
uuid: "7068db6e067942d0bc7a39536ca0902d"
format_stamp: "Formatted at 2026-09-24 11:38:50 on dist-test-slave-cq7c"
server_key: "703356c79d842a0e7e159ecb76edaab1"
server_key_iv: "5695e6573838a6674ef37c5e217aecbb"
server_key_version: "encryptionkey@0"
I20260924 11:38:50.710165 25060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:38:50.725131 25060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:38:50.725919 25060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:38:50.726183 25060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:38:50.727109 25060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:38:50.728542 25060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:38:50.728657 25060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:50.728780 25060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:38:50.728861 25060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:50.762854 25060 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.3:39627
I20260924 11:38:50.762909 25186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.3:39627 every 8 connection(s)
I20260924 11:38:50.764179 25060 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
I20260924 11:38:50.766700 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 25060
I20260924 11:38:50.766877 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance
I20260924 11:38:50.767304 6992 external_mini_cluster.cc:1428] Setting key 5a197cedb7ae0024543fb4e15cc7809b
I20260924 11:38:50.777288 24928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:50.766695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:49591 (local address 127.6.212.62:43633)
0924 11:38:50.766926 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:50.766932 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:50.768543 (+ 1611us) server_negotiation.cc:407] Connection header received
0924 11:38:50.770775 (+ 2232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:50.770779 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:50.770847 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:50.770938 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:38:50.772067 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:50.772809 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:50.774443 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:50.774963 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:50.775063 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:50.776501 (+ 1438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:38:50.776506 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:38:50.776509 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:38:50.776528 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:38:50.776562 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:38:50.776574 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:38:50.776668 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:50.776958 (+ 290us) server_negotiation.cc:299] Negotiation successful
0924 11:38:50.777049 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 11:38:50.778234 25189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:50.767231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:43633 (local address 127.6.212.3:49591)
0924 11:38:50.768355 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:50.768411 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 11:38:50.770540 (+ 2129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:38:50.771197 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:38:50.771205 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:38:50.771295 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:38:50.771928 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:50.771945 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:50.772997 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:50.773001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:50.774254 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:50.774268 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:50.775181 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:50.775189 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:50.775476 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:50.776176 (+ 700us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:38:50.776178 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:38:50.776217 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:38:50.776296 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:38:50.776665 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:38:50.776674 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:38:50.776679 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 11:38:50.776820 (+ 141us) client_negotiation.cc:240] Negotiation successful
0924 11:38:50.777173 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":358,"threads_started":1}
I20260924 11:38:50.779690 25187 heartbeater.cc:347] Connected to a master server at 127.6.212.62:43633
I20260924 11:38:50.780045 25187 heartbeater.cc:464] Registering TS with master...
I20260924 11:38:50.780824 25187 heartbeater.cc:511] Master 127.6.212.62:43633 requested a full tablet report, sending...
I20260924 11:38:50.782382 24740 ts_manager.cc:194] Registered new tserver with Master: 7068db6e067942d0bc7a39536ca0902d (127.6.212.3:39627)
I20260924 11:38:50.783236 24740 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.212.3:49591
I20260924 11:38:50.794947 6992 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:38:50.804601 24928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:50.799146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33518 (local address 127.6.212.62:43633)
0924 11:38:50.799345 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:50.799351 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:50.799763 (+ 412us) server_negotiation.cc:407] Connection header received
0924 11:38:50.799901 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:50.799904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:50.799974 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:50.800053 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:38:50.800507 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:50.801195 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:50.802527 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:50.802927 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:50.803014 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:50.804046 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:38:50.804054 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:38:50.804058 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:38:50.804078 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:38:50.804132 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:38:50.804148 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:38:50.804236 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:50.804314 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 11:38:50.804383 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 11:38:50.808074 24740 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33518:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:38:50.814833 24740 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:38:50.853829 25200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:50.826560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.3:39627 (local address 127.0.0.1:45560)
0924 11:38:50.842459 (+ 15899us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:50.842503 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 11:38:50.842736 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:38:50.843854 (+ 1118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:38:50.843866 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:38:50.843920 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:38:50.844393 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:50.844408 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:50.847235 (+ 2827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:50.847243 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:50.850217 (+ 2974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:50.850249 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:50.852673 (+ 2424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:50.852683 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:50.852997 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:50.853076 (+ 79us) client_negotiation.cc:769] Sending connection context
0924 11:38:50.853219 (+ 143us) client_negotiation.cc:240] Negotiation successful
0924 11:38:50.853429 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15585,"mutex_wait_us":23,"thread_start_us":16979,"threads_started":1}
I20260924 11:38:50.855589 25201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:50.826752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45560 (local address 127.6.212.3:39627)
0924 11:38:50.841919 (+ 15167us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:50.841932 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:50.842623 (+ 691us) server_negotiation.cc:407] Connection header received
0924 11:38:50.843013 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:50.843025 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:50.843340 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:50.844643 (+ 1303us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:38:50.845139 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:50.847046 (+ 1907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:50.850624 (+ 3578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:50.852437 (+ 1813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:50.854639 (+ 2202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:50.854929 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:50.854999 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 11:38:50.855204 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14919,"thread_start_us":14741,"threads_started":1}
I20260924 11:38:50.860286 25203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:50.829046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.2:39635 (local address 127.0.0.1:43660)
0924 11:38:50.850491 (+ 21445us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:50.850511 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 11:38:50.850631 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:38:50.852610 (+ 1979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:38:50.852616 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:38:50.852653 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:38:50.853128 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:50.853139 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:50.855653 (+ 2514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:50.855658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:50.857414 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:50.857426 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:50.859439 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:50.859445 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:50.859645 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:50.859680 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 11:38:50.859769 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 11:38:50.859910 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21302,"mutex_wait_us":18,"thread_start_us":21146,"threads_started":1}
I20260924 11:38:50.860806 25204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:50.829434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43660 (local address 127.6.212.2:39635)
0924 11:38:50.851955 (+ 22521us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:50.851965 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:50.851998 (+ 33us) server_negotiation.cc:407] Connection header received
0924 11:38:50.852105 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:50.852114 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:50.852416 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:50.852632 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:38:50.853459 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:50.855467 (+ 2008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:50.857596 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:50.858901 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:50.859037 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:50.859280 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:50.860263 (+ 983us) server_negotiation.cc:299] Negotiation successful
0924 11:38:50.860478 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22338,"thread_start_us":22212,"threads_started":1}
I20260924 11:38:50.860137 25122 tablet_service.cc:1467] Processing CreateTablet for tablet 65d026e407224995b198604571d22d31 (DEFAULT_TABLE table=test-table [id=8f36fbe6de494e4fb029a639d4a6e989]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:38:50.861729 25122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65d026e407224995b198604571d22d31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:38:50.863585 25199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:50.826627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:43803 (local address 127.0.0.1:53240)
0924 11:38:50.843553 (+ 16926us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:50.843578 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 11:38:50.843727 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:38:50.855769 (+ 12042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:38:50.855776 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:38:50.855820 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:38:50.856255 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:50.856265 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:50.858415 (+ 2150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:50.858421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:50.860715 (+ 2294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:50.860729 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:50.862669 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:50.862677 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:50.862867 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:50.862904 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 11:38:50.863020 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 11:38:50.863164 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16697,"mutex_wait_us":1,"thread_start_us":15382,"threads_started":1}
I20260924 11:38:50.863624 25202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:50.827261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53240 (local address 127.6.212.1:43803)
0924 11:38:50.854874 (+ 27613us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:50.854885 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:50.854929 (+ 44us) server_negotiation.cc:407] Connection header received
0924 11:38:50.855028 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:50.855038 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:50.855286 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:50.855511 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:38:50.856502 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:50.858260 (+ 1758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:50.861006 (+ 2746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:50.862446 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:50.862588 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:50.862861 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:50.863069 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 11:38:50.863227 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25790,"thread_start_us":28464,"threads_started":1}
I20260924 11:38:50.864694 24992 tablet_service.cc:1467] Processing CreateTablet for tablet 65d026e407224995b198604571d22d31 (DEFAULT_TABLE table=test-table [id=8f36fbe6de494e4fb029a639d4a6e989]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:38:50.866108 24992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65d026e407224995b198604571d22d31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:38:50.868495 24861 tablet_service.cc:1467] Processing CreateTablet for tablet 65d026e407224995b198604571d22d31 (DEFAULT_TABLE table=test-table [id=8f36fbe6de494e4fb029a639d4a6e989]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:38:50.869946 24861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65d026e407224995b198604571d22d31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:38:50.873845 24926 heartbeater.cc:503] Master 127.6.212.62:43633 was elected leader, sending a full tablet report...
I20260924 11:38:50.877004 25205 tablet_bootstrap.cc:492] T 65d026e407224995b198604571d22d31 P 7068db6e067942d0bc7a39536ca0902d: Bootstrap starting.
I20260924 11:38:50.880844 25206 tablet_bootstrap.cc:492] T 65d026e407224995b198604571d22d31 P b77a13d463944ae3813c978ea242c9c4: Bootstrap starting.
I20260924 11:38:50.881706 25205 tablet_bootstrap.cc:654] T 65d026e407224995b198604571d22d31 P 7068db6e067942d0bc7a39536ca0902d: Neither blocks nor log segments found. Creating new log.
I20260924 11:38:50.883277 25205 log.cc:824] T 65d026e407224995b198604571d22d31 P 7068db6e067942d0bc7a39536ca0902d: Log is configured to *not* fsync() on all Append() calls
I20260924 11:38:50.884305 25207 tablet_bootstrap.cc:492] T 65d026e407224995b198604571d22d31 P 0b406939cf3a4be29d4b5a6a8c176b1c: Bootstrap starting.
I20260924 11:38:50.885603 25206 tablet_bootstrap.cc:654] T 65d026e407224995b198604571d22d31 P b77a13d463944ae3813c978ea242c9c4: Neither blocks nor log segments found. Creating new log.
I20260924 11:38:50.887290 25206 log.cc:824] T 65d026e407224995b198604571d22d31 P b77a13d463944ae3813c978ea242c9c4: Log is configured to *not* fsync() on all Append() calls
I20260924 11:38:50.889616 25207 tablet_bootstrap.cc:654] T 65d026e407224995b198604571d22d31 P 0b406939cf3a4be29d4b5a6a8c176b1c: Neither blocks nor log segments found. Creating new log.
I20260924 11:38:50.891088 25207 log.cc:824] T 65d026e407224995b198604571d22d31 P 0b406939cf3a4be29d4b5a6a8c176b1c: Log is configured to *not* fsync() on all Append() calls
I20260924 11:38:50.894578 25206 tablet_bootstrap.cc:492] T 65d026e407224995b198604571d22d31 P b77a13d463944ae3813c978ea242c9c4: No bootstrap required, opened a new log
I20260924 11:38:50.894982 25206 ts_tablet_manager.cc:1397] T 65d026e407224995b198604571d22d31 P b77a13d463944ae3813c978ea242c9c4: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260924 11:38:50.897298 25207 tablet_bootstrap.cc:492] T 65d026e407224995b198604571d22d31 P 0b406939cf3a4be29d4b5a6a8c176b1c: No bootstrap required, opened a new log
I20260924 11:38:50.898001 25207 ts_tablet_manager.cc:1397] T 65d026e407224995b198604571d22d31 P 0b406939cf3a4be29d4b5a6a8c176b1c: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.005s
I20260924 11:38:50.901199 25205 tablet_bootstrap.cc:492] T 65d026e407224995b198604571d22d31 P 7068db6e067942d0bc7a39536ca0902d: No bootstrap required, opened a new log
I20260924 11:38:50.901523 25205 ts_tablet_manager.cc:1397] T 65d026e407224995b198604571d22d31 P 7068db6e067942d0bc7a39536ca0902d: Time spent bootstrapping tablet: real 0.025s user 0.009s sys 0.012s
I20260924 11:38:50.909008 25207 raft_consensus.cc:348] T 65d026e407224995b198604571d22d31 P 0b406939cf3a4be29d4b5a6a8c176b1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7068db6e067942d0bc7a39536ca0902d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 39627 } } peers { permanent_uuid: "0b406939cf3a4be29d4b5a6a8c176b1c" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 43803 } } peers { permanent_uuid: "b77a13d463944ae3813c978ea242c9c4" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 39635 } }
I20260924 11:38:50.909639 25207 raft_consensus.cc:374] T 65d026e407224995b198604571d22d31 P 0b406939cf3a4be29d4b5a6a8c176b1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:38:50.909785 25207 raft_consensus.cc:729] T 65d026e407224995b198604571d22d31 P 0b406939cf3a4be29d4b5a6a8c176b1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b406939cf3a4be29d4b5a6a8c176b1c, State: Initialized, Role: FOLLOWER
I20260924 11:38:50.910434 25206 raft_consensus.cc:348] T 65d026e407224995b198604571d22d31 P b77a13d463944ae3813c978ea242c9c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7068db6e067942d0bc7a39536ca0902d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 39627 } } peers { permanent_uuid: "0b406939cf3a4be29d4b5a6a8c176b1c" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 43803 } } peers { permanent_uuid: "b77a13d463944ae3813c978ea242c9c4" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 39635 } }
I20260924 11:38:50.910633 25207 consensus_queue.cc:260] T 65d026e407224995b198604571d22d31 P 0b406939cf3a4be29d4b5a6a8c176b1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7068db6e067942d0bc7a39536ca0902d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 39627 } } peers { permanent_uuid: "0b406939cf3a4be29d4b5a6a8c176b1c" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 43803 } } peers { permanent_uuid: "b77a13d463944ae3813c978ea242c9c4" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 39635 } }
I20260924 11:38:50.911028 25206 raft_consensus.cc:374] T 65d026e407224995b198604571d22d31 P b77a13d463944ae3813c978ea242c9c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:38:50.911194 25206 raft_consensus.cc:729] T 65d026e407224995b198604571d22d31 P b77a13d463944ae3813c978ea242c9c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b77a13d463944ae3813c978ea242c9c4, State: Initialized, Role: FOLLOWER
I20260924 11:38:50.911957 25206 consensus_queue.cc:260] T 65d026e407224995b198604571d22d31 P b77a13d463944ae3813c978ea242c9c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7068db6e067942d0bc7a39536ca0902d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 39627 } } peers { permanent_uuid: "0b406939cf3a4be29d4b5a6a8c176b1c" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 43803 } } peers { permanent_uuid: "b77a13d463944ae3813c978ea242c9c4" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 39635 } }
I20260924 11:38:50.912925 25207 ts_tablet_manager.cc:1428] T 65d026e407224995b198604571d22d31 P 0b406939cf3a4be29d4b5a6a8c176b1c: Time spent starting tablet: real 0.015s user 0.012s sys 0.001s
I20260924 11:38:50.913847 25057 heartbeater.cc:503] Master 127.6.212.62:43633 was elected leader, sending a full tablet report...
I20260924 11:38:50.914754 25206 ts_tablet_manager.cc:1428] T 65d026e407224995b198604571d22d31 P b77a13d463944ae3813c978ea242c9c4: Time spent starting tablet: real 0.020s user 0.018s sys 0.000s
I20260924 11:38:50.915802 25205 raft_consensus.cc:348] T 65d026e407224995b198604571d22d31 P 7068db6e067942d0bc7a39536ca0902d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7068db6e067942d0bc7a39536ca0902d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 39627 } } peers { permanent_uuid: "0b406939cf3a4be29d4b5a6a8c176b1c" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 43803 } } peers { permanent_uuid: "b77a13d463944ae3813c978ea242c9c4" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 39635 } }
I20260924 11:38:50.916335 25205 raft_consensus.cc:374] T 65d026e407224995b198604571d22d31 P 7068db6e067942d0bc7a39536ca0902d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:38:50.916456 25205 raft_consensus.cc:729] T 65d026e407224995b198604571d22d31 P 7068db6e067942d0bc7a39536ca0902d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7068db6e067942d0bc7a39536ca0902d, State: Initialized, Role: FOLLOWER
I20260924 11:38:50.917318 25205 consensus_queue.cc:260] T 65d026e407224995b198604571d22d31 P 7068db6e067942d0bc7a39536ca0902d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7068db6e067942d0bc7a39536ca0902d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 39627 } } peers { permanent_uuid: "0b406939cf3a4be29d4b5a6a8c176b1c" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 43803 } } peers { permanent_uuid: "b77a13d463944ae3813c978ea242c9c4" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 39635 } }
I20260924 11:38:50.919644 25205 ts_tablet_manager.cc:1428] T 65d026e407224995b198604571d22d31 P 7068db6e067942d0bc7a39536ca0902d: Time spent starting tablet: real 0.018s user 0.010s sys 0.004s
I20260924 11:38:50.920533 25187 heartbeater.cc:503] Master 127.6.212.62:43633 was elected leader, sending a full tablet report...
W20260924 11:38:51.015815 25188 tablet.cc:2369] T 65d026e407224995b198604571d22d31 P 7068db6e067942d0bc7a39536ca0902d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:38:51.072798 25058 tablet.cc:2369] T 65d026e407224995b198604571d22d31 P b77a13d463944ae3813c978ea242c9c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:38:51.098094 25212 raft_consensus.cc:482] T 65d026e407224995b198604571d22d31 P b77a13d463944ae3813c978ea242c9c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:38:51.098515 25212 raft_consensus.cc:504] T 65d026e407224995b198604571d22d31 P b77a13d463944ae3813c978ea242c9c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7068db6e067942d0bc7a39536ca0902d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 39627 } } peers { permanent_uuid: "0b406939cf3a4be29d4b5a6a8c176b1c" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 43803 } } peers { permanent_uuid: "b77a13d463944ae3813c978ea242c9c4" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 39635 } }
I20260924 11:38:51.100581 25212 leader_election.cc:288] T 65d026e407224995b198604571d22d31 P b77a13d463944ae3813c978ea242c9c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7068db6e067942d0bc7a39536ca0902d (127.6.212.3:39627), 0b406939cf3a4be29d4b5a6a8c176b1c (127.6.212.1:43803)
W20260924 11:38:51.100972 24927 tablet.cc:2369] T 65d026e407224995b198604571d22d31 P 0b406939cf3a4be29d4b5a6a8c176b1c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:38:51.108577 25202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:51.102246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:37673 (local address 127.6.212.1:43803)
0924 11:38:51.102465 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:51.102469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:51.102484 (+ 15us) server_negotiation.cc:407] Connection header received
0924 11:38:51.102630 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:51.102634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:51.102752 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:51.102854 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:38:51.103650 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:51.104439 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:51.106621 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:51.107525 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:51.107670 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:51.107786 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:51.108202 (+ 416us) server_negotiation.cc:299] Negotiation successful
0924 11:38:51.108330 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 11:38:51.108604 25215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:51.100793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.3:39627 (local address 127.6.212.2:43089)
0924 11:38:51.101976 (+ 1183us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:51.102003 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 11:38:51.102235 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:38:51.102731 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:38:51.102737 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:38:51.102762 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:38:51.103229 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:51.103240 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:51.104467 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:51.104477 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:51.106264 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:51.106281 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:51.107785 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:51.107792 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:51.107973 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:51.107997 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 11:38:51.108071 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 11:38:51.108205 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":990,"mutex_wait_us":7,"spinlock_wait_cycles":25472,"thread_start_us":1128,"threads_started":1}
I20260924 11:38:51.108597 25201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:51.101082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:43089 (local address 127.6.212.3:39627)
0924 11:38:51.101328 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:51.101338 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:51.102107 (+ 769us) server_negotiation.cc:407] Connection header received
0924 11:38:51.102478 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:51.102485 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:51.102586 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:51.102694 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:38:51.103432 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:51.104309 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:51.106480 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:51.107536 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:51.107712 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:51.107860 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:51.108151 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 11:38:51.108301 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 11:38:51.108758 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:51.100943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:43803 (local address 127.6.212.2:37673)
0924 11:38:51.102266 (+ 1323us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:51.102288 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 11:38:51.102495 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:38:51.102909 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:38:51.102913 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:38:51.102951 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:38:51.103432 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:51.103447 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:51.104606 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:51.104611 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:51.106360 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:51.106410 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:51.107771 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:51.107779 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:51.108000 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:51.108026 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 11:38:51.108162 (+ 136us) client_negotiation.cc:240] Negotiation successful
0924 11:38:51.108314 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1178,"mutex_wait_us":1,"thread_start_us":585,"threads_started":1}
I20260924 11:38:51.110000 24881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d026e407224995b198604571d22d31" candidate_uuid: "b77a13d463944ae3813c978ea242c9c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b406939cf3a4be29d4b5a6a8c176b1c" is_pre_election: true
I20260924 11:38:51.110061 25142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d026e407224995b198604571d22d31" candidate_uuid: "b77a13d463944ae3813c978ea242c9c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7068db6e067942d0bc7a39536ca0902d" is_pre_election: true
I20260924 11:38:51.110730 24881 raft_consensus.cc:2454] T 65d026e407224995b198604571d22d31 P 0b406939cf3a4be29d4b5a6a8c176b1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b77a13d463944ae3813c978ea242c9c4 in term 0.
I20260924 11:38:51.110730 25142 raft_consensus.cc:2454] T 65d026e407224995b198604571d22d31 P 7068db6e067942d0bc7a39536ca0902d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b77a13d463944ae3813c978ea242c9c4 in term 0.
I20260924 11:38:51.111611 24947 leader_election.cc:302] T 65d026e407224995b198604571d22d31 P b77a13d463944ae3813c978ea242c9c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0b406939cf3a4be29d4b5a6a8c176b1c, 7068db6e067942d0bc7a39536ca0902d, b77a13d463944ae3813c978ea242c9c4; no voters:
I20260924 11:38:51.112167 25212 raft_consensus.cc:2781] T 65d026e407224995b198604571d22d31 P b77a13d463944ae3813c978ea242c9c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:38:51.112340 25212 raft_consensus.cc:482] T 65d026e407224995b198604571d22d31 P b77a13d463944ae3813c978ea242c9c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:38:51.112453 25212 raft_consensus.cc:3037] T 65d026e407224995b198604571d22d31 P b77a13d463944ae3813c978ea242c9c4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:38:51.114725 25212 raft_consensus.cc:504] T 65d026e407224995b198604571d22d31 P b77a13d463944ae3813c978ea242c9c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7068db6e067942d0bc7a39536ca0902d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 39627 } } peers { permanent_uuid: "0b406939cf3a4be29d4b5a6a8c176b1c" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 43803 } } peers { permanent_uuid: "b77a13d463944ae3813c978ea242c9c4" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 39635 } }
I20260924 11:38:51.115316 25212 leader_election.cc:288] T 65d026e407224995b198604571d22d31 P b77a13d463944ae3813c978ea242c9c4 [CANDIDATE]: Term 1 election: Requested vote from peers 7068db6e067942d0bc7a39536ca0902d (127.6.212.3:39627), 0b406939cf3a4be29d4b5a6a8c176b1c (127.6.212.1:43803)
I20260924 11:38:51.115845 25142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d026e407224995b198604571d22d31" candidate_uuid: "b77a13d463944ae3813c978ea242c9c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7068db6e067942d0bc7a39536ca0902d"
I20260924 11:38:51.115935 24881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d026e407224995b198604571d22d31" candidate_uuid: "b77a13d463944ae3813c978ea242c9c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b406939cf3a4be29d4b5a6a8c176b1c"
I20260924 11:38:51.116110 25142 raft_consensus.cc:3037] T 65d026e407224995b198604571d22d31 P 7068db6e067942d0bc7a39536ca0902d [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:38:51.116154 24881 raft_consensus.cc:3037] T 65d026e407224995b198604571d22d31 P 0b406939cf3a4be29d4b5a6a8c176b1c [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:38:51.118371 24881 raft_consensus.cc:2454] T 65d026e407224995b198604571d22d31 P 0b406939cf3a4be29d4b5a6a8c176b1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b77a13d463944ae3813c978ea242c9c4 in term 1.
I20260924 11:38:51.118371 25142 raft_consensus.cc:2454] T 65d026e407224995b198604571d22d31 P 7068db6e067942d0bc7a39536ca0902d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b77a13d463944ae3813c978ea242c9c4 in term 1.
I20260924 11:38:51.118835 24947 leader_election.cc:302] T 65d026e407224995b198604571d22d31 P b77a13d463944ae3813c978ea242c9c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b406939cf3a4be29d4b5a6a8c176b1c, b77a13d463944ae3813c978ea242c9c4; no voters:
I20260924 11:38:51.119431 25212 raft_consensus.cc:2781] T 65d026e407224995b198604571d22d31 P b77a13d463944ae3813c978ea242c9c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:38:51.119943 25212 raft_consensus.cc:686] T 65d026e407224995b198604571d22d31 P b77a13d463944ae3813c978ea242c9c4 [term 1 LEADER]: Becoming Leader. State: Replica: b77a13d463944ae3813c978ea242c9c4, State: Running, Role: LEADER
I20260924 11:38:51.120440 25212 consensus_queue.cc:237] T 65d026e407224995b198604571d22d31 P b77a13d463944ae3813c978ea242c9c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7068db6e067942d0bc7a39536ca0902d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 39627 } } peers { permanent_uuid: "0b406939cf3a4be29d4b5a6a8c176b1c" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 43803 } } peers { permanent_uuid: "b77a13d463944ae3813c978ea242c9c4" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 39635 } }
I20260924 11:38:51.125139 24740 catalog_manager.cc:5495] T 65d026e407224995b198604571d22d31 P b77a13d463944ae3813c978ea242c9c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to b77a13d463944ae3813c978ea242c9c4 (127.6.212.2). New cstate: current_term: 1 leader_uuid: "b77a13d463944ae3813c978ea242c9c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7068db6e067942d0bc7a39536ca0902d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 39627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b406939cf3a4be29d4b5a6a8c176b1c" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 43803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b77a13d463944ae3813c978ea242c9c4" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 39635 } health_report { overall_health: HEALTHY } } }
I20260924 11:38:51.156519 25204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:51.150530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43666 (local address 127.6.212.2:39635)
0924 11:38:51.150772 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:51.150780 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:51.150800 (+ 20us) server_negotiation.cc:407] Connection header received
0924 11:38:51.150917 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:51.150921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:51.151012 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:51.151178 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:38:51.151619 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:51.152363 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:51.154030 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:51.154515 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:51.154681 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:51.155248 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:38:51.155869 (+ 621us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:38:51.156067 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:51.156122 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 11:38:51.156209 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260924 11:38:51.165319 24881 raft_consensus.cc:1261] T 65d026e407224995b198604571d22d31 P 0b406939cf3a4be29d4b5a6a8c176b1c [term 1 FOLLOWER]: Refusing update from remote peer b77a13d463944ae3813c978ea242c9c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:38:51.165318 25142 raft_consensus.cc:1261] T 65d026e407224995b198604571d22d31 P 7068db6e067942d0bc7a39536ca0902d [term 1 FOLLOWER]: Refusing update from remote peer b77a13d463944ae3813c978ea242c9c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:38:51.166356 25216 consensus_queue.cc:1035] T 65d026e407224995b198604571d22d31 P b77a13d463944ae3813c978ea242c9c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b406939cf3a4be29d4b5a6a8c176b1c" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 43803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:38:51.166705 25212 consensus_queue.cc:1035] T 65d026e407224995b198604571d22d31 P b77a13d463944ae3813c978ea242c9c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7068db6e067942d0bc7a39536ca0902d" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 39627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:38:51.179528 25219 mvcc.cc:205] Tried to move back new op lower bound from 7332863718042476544 to 7332863717873373184. Current Snapshot: MvccSnapshot[applied={T|T < 7332863718042476544}]
I20260924 11:38:51.194047 24739 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:38:51.194414 24739 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:38:51.198606 24739 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9cb443dbf3f44831b10bf7f86705559c: Sending DeleteTablet for 3 replicas of tablet 65d026e407224995b198604571d22d31
I20260924 11:38:51.199997 25122 tablet_service.cc:1514] Processing DeleteTablet for tablet 65d026e407224995b198604571d22d31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:38:51 UTC) from {username='slave'} at 127.0.0.1:45560
I20260924 11:38:51.199993 24861 tablet_service.cc:1514] Processing DeleteTablet for tablet 65d026e407224995b198604571d22d31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:38:51 UTC) from {username='slave'} at 127.0.0.1:53240
I20260924 11:38:51.200969 25229 tablet_replica.cc:330] T 65d026e407224995b198604571d22d31 P 7068db6e067942d0bc7a39536ca0902d: stopping tablet replica
I20260924 11:38:51.201135 24992 tablet_service.cc:1514] Processing DeleteTablet for tablet 65d026e407224995b198604571d22d31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:38:51 UTC) from {username='slave'} at 127.0.0.1:43660
I20260924 11:38:51.201529 25229 raft_consensus.cc:2229] T 65d026e407224995b198604571d22d31 P 7068db6e067942d0bc7a39536ca0902d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:38:51.202098 25229 raft_consensus.cc:2258] T 65d026e407224995b198604571d22d31 P 7068db6e067942d0bc7a39536ca0902d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:38:51.202975 25230 tablet_replica.cc:330] T 65d026e407224995b198604571d22d31 P 0b406939cf3a4be29d4b5a6a8c176b1c: stopping tablet replica
I20260924 11:38:51.203184 25231 tablet_replica.cc:330] T 65d026e407224995b198604571d22d31 P b77a13d463944ae3813c978ea242c9c4: stopping tablet replica
I20260924 11:38:51.203667 25231 raft_consensus.cc:2229] T 65d026e407224995b198604571d22d31 P b77a13d463944ae3813c978ea242c9c4 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:38:51.203925 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 24799
I20260924 11:38:51.204336 25230 raft_consensus.cc:2229] T 65d026e407224995b198604571d22d31 P 0b406939cf3a4be29d4b5a6a8c176b1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:38:51.204463 25231 raft_consensus.cc:2258] T 65d026e407224995b198604571d22d31 P b77a13d463944ae3813c978ea242c9c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:38:51.205201 25230 raft_consensus.cc:2258] T 65d026e407224995b198604571d22d31 P 0b406939cf3a4be29d4b5a6a8c176b1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:38:51.207005 25231 ts_tablet_manager.cc:1905] T 65d026e407224995b198604571d22d31 P b77a13d463944ae3813c978ea242c9c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:38:51.207461 25229 ts_tablet_manager.cc:1905] T 65d026e407224995b198604571d22d31 P 7068db6e067942d0bc7a39536ca0902d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:38:51.207554 25230 ts_tablet_manager.cc:1905] T 65d026e407224995b198604571d22d31 P 0b406939cf3a4be29d4b5a6a8c176b1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:38:51.214540 25229 ts_tablet_manager.cc:1918] T 65d026e407224995b198604571d22d31 P 7068db6e067942d0bc7a39536ca0902d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:38:51.214686 25231 ts_tablet_manager.cc:1918] T 65d026e407224995b198604571d22d31 P b77a13d463944ae3813c978ea242c9c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:38:51.214784 25229 log.cc:1196] T 65d026e407224995b198604571d22d31 P 7068db6e067942d0bc7a39536ca0902d: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-2/wal/wals/65d026e407224995b198604571d22d31
I20260924 11:38:51.214912 25231 log.cc:1196] T 65d026e407224995b198604571d22d31 P b77a13d463944ae3813c978ea242c9c4: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-1/wal/wals/65d026e407224995b198604571d22d31
I20260924 11:38:51.215602 25231 ts_tablet_manager.cc:1939] T 65d026e407224995b198604571d22d31 P b77a13d463944ae3813c978ea242c9c4: Deleting consensus metadata
I20260924 11:38:51.215739 25229 ts_tablet_manager.cc:1939] T 65d026e407224995b198604571d22d31 P 7068db6e067942d0bc7a39536ca0902d: Deleting consensus metadata
I20260924 11:38:51.217597 24724 catalog_manager.cc:4841] TS b77a13d463944ae3813c978ea242c9c4 (127.6.212.2:39635): tablet 65d026e407224995b198604571d22d31 (table test-table [id=8f36fbe6de494e4fb029a639d4a6e989]) successfully deleted
I20260924 11:38:51.217630 24727 catalog_manager.cc:4841] TS 7068db6e067942d0bc7a39536ca0902d (127.6.212.3:39627): tablet 65d026e407224995b198604571d22d31 (table test-table [id=8f36fbe6de494e4fb029a639d4a6e989]) successfully deleted
I20260924 11:38:51.217149 25230 ts_tablet_manager.cc:1918] T 65d026e407224995b198604571d22d31 P 0b406939cf3a4be29d4b5a6a8c176b1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:38:51.218114 25230 log.cc:1196] T 65d026e407224995b198604571d22d31 P 0b406939cf3a4be29d4b5a6a8c176b1c: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790249617057452-6992-0/minicluster-data/ts-0/wal/wals/65d026e407224995b198604571d22d31
I20260924 11:38:51.218814 25230 ts_tablet_manager.cc:1939] T 65d026e407224995b198604571d22d31 P 0b406939cf3a4be29d4b5a6a8c176b1c: Deleting consensus metadata
I20260924 11:38:51.220947 24726 catalog_manager.cc:4841] TS 0b406939cf3a4be29d4b5a6a8c176b1c (127.6.212.1:43803): tablet 65d026e407224995b198604571d22d31 (table test-table [id=8f36fbe6de494e4fb029a639d4a6e989]) successfully deleted
I20260924 11:38:51.230844 25202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:51.205450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53254 (local address 127.6.212.1:43803)
0924 11:38:51.205650 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:51.205657 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:51.219591 (+ 13934us) server_negotiation.cc:407] Connection header received
0924 11:38:51.219843 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:51.219848 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:51.219955 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:51.220067 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:38:51.220625 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:51.221886 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:51.223292 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:51.223943 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:51.224078 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:51.229065 (+ 4987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:38:51.229076 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:38:51.229097 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:38:51.229953 (+ 856us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:38:51.230029 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:38:51.230059 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:38:51.230184 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:51.230330 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 11:38:51.230455 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 11:38:51.231684 24921 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:38:51.385066 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 24799
I20260924 11:38:51.409554 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 24930
I20260924 11:38:51.417927 25204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:51.410449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43672 (local address 127.6.212.2:39635)
0924 11:38:51.410723 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:51.410731 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:51.410760 (+ 29us) server_negotiation.cc:407] Connection header received
0924 11:38:51.410949 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:51.410954 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:51.411045 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:51.411181 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:38:51.411688 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:51.412606 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:51.414324 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:51.414792 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:51.414966 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:51.416334 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:38:51.416347 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:38:51.416364 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:38:51.417030 (+ 666us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:38:51.417130 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:38:51.417155 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:38:51.417285 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:51.417452 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 11:38:51.417565 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260924 11:38:51.418690 25052 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:38:51.534423 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 24930
I20260924 11:38:51.553141 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 25060
I20260924 11:38:51.560545 25201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:51.554030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45574 (local address 127.6.212.3:39627)
0924 11:38:51.554224 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:51.554230 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:51.554289 (+ 59us) server_negotiation.cc:407] Connection header received
0924 11:38:51.554525 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:51.554530 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:51.554617 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:51.554731 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:38:51.555148 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:51.555943 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:51.557285 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:51.557682 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:51.557825 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:51.559034 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:38:51.559044 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:38:51.559059 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:38:51.559766 (+ 707us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:38:51.559818 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:38:51.559838 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:38:51.559968 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:51.560108 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 11:38:51.560218 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 11:38:51.561194 25182 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:38:51.651047 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 25060
I20260924 11:38:51.668048 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 24709
I20260924 11:38:51.669173 24770 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:38:51.782151 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 24709
2026-09-24T11:38:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3039 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:38:51 dist-test-slave-cq7c krb5kdc[25239](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:38:51 dist-test-slave-cq7c krb5kdc[25239](info): set up 2 sockets
Sep 24 11:38:51 dist-test-slave-cq7c krb5kdc[25239](info): commencing operation
krb5kdc: starting...
W20260924 11:38:52.903793 6992 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Sep 24 11:38:53 dist-test-slave-cq7c krb5kdc[25239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:38:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:38:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.212.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
I20260924 11:38:53.175510 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:43561
--webserver_interface=127.6.212.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:41023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:43561
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:38:53.527746 25252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:38:53.528129 25252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:38:53.528298 25252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:38:53.535998 25252 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:38:53.536190 25252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:38:53.536289 25252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:38:53.536392 25252 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:38:53.536477 25252 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:38:53.547058 25252 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:41023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:43561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:43561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.212.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:38:53.548585 25252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:38:53.550120 25252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:38:53.563225 25257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:38:53.563310 25258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:38:53.563750 25260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:38:53.564286 25252 server_base.cc:1034] running on GCE node
I20260924 11:38:53.565196 25252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:38:53.566774 25252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:38:53.568002 25252 hybrid_clock.cc:648] HybridClock initialized: now 1790249933567923 us; error 58 us; skew 500 ppm
I20260924 11:38:53.568440 25252 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:38:53 dist-test-slave-cq7c krb5kdc[25239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249933, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:38:53.572073 25252 init.cc:377] Logged in from keytab as kudu/127.6.212.62@KRBTEST.COM (short username kudu)
I20260924 11:38:53.573945 25252 webserver.cc:460] Webserver started at http://127.6.212.62:40757/ using document root <none> and password file <none>
I20260924 11:38:53.574764 25252 fs_manager.cc:359] Metadata directory not provided
I20260924 11:38:53.574952 25252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:38:53.575336 25252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:38:53.579058 25252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "ab5b92dab57e4c148f77c2481a280c07"
format_stamp: "Formatted at 2026-09-24 11:38:53 on dist-test-slave-cq7c"
server_key: "a82c64184739305b4a3d847187a57f38"
server_key_iv: "14240be706a049a9f7453607955be6ea"
server_key_version: "encryptionkey@0"
I20260924 11:38:53.580183 25252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "ab5b92dab57e4c148f77c2481a280c07"
format_stamp: "Formatted at 2026-09-24 11:38:53 on dist-test-slave-cq7c"
server_key: "a82c64184739305b4a3d847187a57f38"
server_key_iv: "14240be706a049a9f7453607955be6ea"
server_key_version: "encryptionkey@0"
I20260924 11:38:53.586984 25252 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 11:38:53.591109 25267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:53.593011 25252 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 11:38:53.593257 25252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "ab5b92dab57e4c148f77c2481a280c07"
format_stamp: "Formatted at 2026-09-24 11:38:53 on dist-test-slave-cq7c"
server_key: "a82c64184739305b4a3d847187a57f38"
server_key_iv: "14240be706a049a9f7453607955be6ea"
server_key_version: "encryptionkey@0"
I20260924 11:38:53.593498 25252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:38:53.611171 25252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:38:53.615849 25252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:38:53.616170 25252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:38:53.635437 25252 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.62:43561
I20260924 11:38:53.635540 25318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.62:43561 every 8 connection(s)
I20260924 11:38:53.636924 25252 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:38:53.640969 25319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:38:53.642819 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 25252
I20260924 11:38:53.643023 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:38:53.643604 6992 external_mini_cluster.cc:1428] Setting key 82064e326d131a716017ae5bad8f5512
I20260924 11:38:53.656176 25319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab5b92dab57e4c148f77c2481a280c07: Bootstrap starting.
Sep 24 11:38:53 dist-test-slave-cq7c krb5kdc[25239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:38:53.660393 25319 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab5b92dab57e4c148f77c2481a280c07: Neither blocks nor log segments found. Creating new log.
I20260924 11:38:53.661684 25319 log.cc:824] T 00000000000000000000000000000000 P ab5b92dab57e4c148f77c2481a280c07: Log is configured to *not* fsync() on all Append() calls
I20260924 11:38:53.665819 25319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab5b92dab57e4c148f77c2481a280c07: No bootstrap required, opened a new log
I20260924 11:38:53.671111 25322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:53.645638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41666 (local address 127.6.212.62:43561)
0924 11:38:53.646500 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:53.646515 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:53.646567 (+ 52us) server_negotiation.cc:407] Connection header received
0924 11:38:53.648351 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:53.648376 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:53.648809 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:53.649138 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:38:53.651147 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:53.652816 (+ 1669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:53.654983 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:53.655875 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:53.656044 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:53.661147 (+ 5103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:38:53.661159 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:38:53.661178 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:38:53.661230 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:38:53.665692 (+ 4462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:53.666609 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:53.666617 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:53.666626 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:53.666871 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:53.667392 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:53.667397 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:53.667400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:53.668152 (+ 752us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:38:53.668468 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:53.669024 (+ 556us) server_negotiation.cc:299] Negotiation successful
0924 11:38:53.669642 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":417,"threads_started":1}
I20260924 11:38:53.676332 25319 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab5b92dab57e4c148f77c2481a280c07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab5b92dab57e4c148f77c2481a280c07" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43561 } }
I20260924 11:38:53.676779 25319 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab5b92dab57e4c148f77c2481a280c07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:38:53.676862 25319 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab5b92dab57e4c148f77c2481a280c07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5b92dab57e4c148f77c2481a280c07, State: Initialized, Role: FOLLOWER
I20260924 11:38:53.677500 25319 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab5b92dab57e4c148f77c2481a280c07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab5b92dab57e4c148f77c2481a280c07" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43561 } }
I20260924 11:38:53.677737 25319 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab5b92dab57e4c148f77c2481a280c07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:38:53.677899 25319 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab5b92dab57e4c148f77c2481a280c07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:38:53.678053 25319 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab5b92dab57e4c148f77c2481a280c07 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:38:53.680922 25319 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab5b92dab57e4c148f77c2481a280c07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab5b92dab57e4c148f77c2481a280c07" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43561 } }
I20260924 11:38:53.681510 25319 leader_election.cc:302] T 00000000000000000000000000000000 P ab5b92dab57e4c148f77c2481a280c07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab5b92dab57e4c148f77c2481a280c07; no voters:
I20260924 11:38:53.682253 25319 leader_election.cc:288] T 00000000000000000000000000000000 P ab5b92dab57e4c148f77c2481a280c07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:38:53.682462 25324 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab5b92dab57e4c148f77c2481a280c07 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:38:53.683411 25324 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab5b92dab57e4c148f77c2481a280c07 [term 1 LEADER]: Becoming Leader. State: Replica: ab5b92dab57e4c148f77c2481a280c07, State: Running, Role: LEADER
I20260924 11:38:53.684123 25324 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab5b92dab57e4c148f77c2481a280c07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab5b92dab57e4c148f77c2481a280c07" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43561 } }
I20260924 11:38:53.684867 25319 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab5b92dab57e4c148f77c2481a280c07 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:38:53.687332 25326 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab5b92dab57e4c148f77c2481a280c07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab5b92dab57e4c148f77c2481a280c07. Latest consensus state: current_term: 1 leader_uuid: "ab5b92dab57e4c148f77c2481a280c07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab5b92dab57e4c148f77c2481a280c07" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43561 } } }
I20260924 11:38:53.687923 25326 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab5b92dab57e4c148f77c2481a280c07 [sys.catalog]: This master's current role is: LEADER
I20260924 11:38:53.688962 25325 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab5b92dab57e4c148f77c2481a280c07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab5b92dab57e4c148f77c2481a280c07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab5b92dab57e4c148f77c2481a280c07" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43561 } } }
I20260924 11:38:53.689682 25325 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab5b92dab57e4c148f77c2481a280c07 [sys.catalog]: This master's current role is: LEADER
I20260924 11:38:53.691411 25331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:38:53.697665 25331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:38:53.716861 25331 catalog_manager.cc:1347] Generated new cluster ID: 992e916a34e049eb8c9444b0b9fb7187
I20260924 11:38:53.717084 25331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:38:53.734910 25331 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:38:53.736971 25331 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:38:53.752053 25331 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab5b92dab57e4c148f77c2481a280c07: Generated new TSK 0
I20260924 11:38:53.753190 25331 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.212.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab.
Entry for principal kudu/127.6.212.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab.
I20260924 11:38:53.828155 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.1:0
--local_ip_for_outbound_sockets=127.6.212.1
--webserver_interface=127.6.212.1
--webserver_port=0
--tserver_master_addrs=127.6.212.62:43561
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab
--principal=kudu/127.6.212.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:41023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:38:54.176460 25345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:38:54.176753 25345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:38:54.176921 25345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:38:54.184381 25345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:38:54.184563 25345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:38:54.184773 25345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.1
I20260924 11:38:54.195489 25345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:41023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab
--principal=kudu/127.6.212.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.212.1
--webserver_port=0
--tserver_master_addrs=127.6.212.62:43561
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.1
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:38:54.196871 25345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:38:54.198344 25345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:38:54.210945 25350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:38:54.212332 25351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:38:54.212736 25353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:38:54.213270 25345 server_base.cc:1034] running on GCE node
I20260924 11:38:54.213894 25345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:38:54.214874 25345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:38:54.216173 25345 hybrid_clock.cc:648] HybridClock initialized: now 1790249934216100 us; error 56 us; skew 500 ppm
I20260924 11:38:54.216573 25345 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:38:54 dist-test-slave-cq7c krb5kdc[25239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249934, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:38:54.220451 25345 init.cc:377] Logged in from keytab as kudu/127.6.212.1@KRBTEST.COM (short username kudu)
I20260924 11:38:54.222172 25345 webserver.cc:460] Webserver started at http://127.6.212.1:42511/ using document root <none> and password file <none>
I20260924 11:38:54.223006 25345 fs_manager.cc:359] Metadata directory not provided
I20260924 11:38:54.223176 25345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:38:54.223606 25345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:38:54.226974 25345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-0/data/instance:
uuid: "2ffc01d8c4714548b1627fc67fc60edb"
format_stamp: "Formatted at 2026-09-24 11:38:54 on dist-test-slave-cq7c"
server_key: "cb4148b429e2faec8e058922f5a62204"
server_key_iv: "44350cb7f2f5c3707a68cab887b8c4ea"
server_key_version: "encryptionkey@0"
I20260924 11:38:54.227900 25345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance:
uuid: "2ffc01d8c4714548b1627fc67fc60edb"
format_stamp: "Formatted at 2026-09-24 11:38:54 on dist-test-slave-cq7c"
server_key: "cb4148b429e2faec8e058922f5a62204"
server_key_iv: "44350cb7f2f5c3707a68cab887b8c4ea"
server_key_version: "encryptionkey@0"
I20260924 11:38:54.233652 25345 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 11:38:54.237661 25360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:54.239261 25345 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 11:38:54.239548 25345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-0/wal
uuid: "2ffc01d8c4714548b1627fc67fc60edb"
format_stamp: "Formatted at 2026-09-24 11:38:54 on dist-test-slave-cq7c"
server_key: "cb4148b429e2faec8e058922f5a62204"
server_key_iv: "44350cb7f2f5c3707a68cab887b8c4ea"
server_key_version: "encryptionkey@0"
I20260924 11:38:54.239786 25345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:38:54.286579 25345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:38:54.291623 25345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:38:54.291973 25345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:38:54.293012 25345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:38:54.294512 25345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:38:54.294631 25345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:54.294798 25345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:38:54.294888 25345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:54.330873 25345 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.1:45239
I20260924 11:38:54.330904 25472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.1:45239 every 8 connection(s)
I20260924 11:38:54.332309 25345 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
I20260924 11:38:54.335932 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 25345
I20260924 11:38:54.336176 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance
I20260924 11:38:54.336715 6992 external_mini_cluster.cc:1428] Setting key e16b629e03c8d0c6a42fa308df8c082e
Sep 24 11:38:54 dist-test-slave-cq7c krb5kdc[25239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249934, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:38:54.354708 25475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:54.334978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:48893 (local address 127.6.212.62:43561)
0924 11:38:54.335501 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:54.335506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:54.336743 (+ 1237us) server_negotiation.cc:407] Connection header received
0924 11:38:54.339269 (+ 2526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:54.339274 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:54.339385 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:54.339531 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:38:54.341023 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:54.341742 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:54.343419 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:54.343902 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:54.344039 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:54.347549 (+ 3510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:38:54.347554 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:38:54.347559 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:38:54.347596 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:38:54.351282 (+ 3686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:54.352350 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:54.352358 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:54.352364 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:54.352502 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:54.352947 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:54.352952 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:54.352955 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:54.353227 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:38:54.353376 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:54.354117 (+ 741us) server_negotiation.cc:299] Negotiation successful
0924 11:38:54.354337 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":240,"threads_started":1}
I20260924 11:38:54.355885 25476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:54.335699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:43561 (local address 127.6.212.1:48893)
0924 11:38:54.336577 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:54.336616 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 11:38:54.339010 (+ 2394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:38:54.339768 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:38:54.339776 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:38:54.340290 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:38:54.340859 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:54.340877 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:54.341897 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:54.341903 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:54.343236 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:54.343248 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:54.344126 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:54.344134 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:54.344440 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:54.345141 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:38:54.345164 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:38:54.347300 (+ 2136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:38:54.351486 (+ 4186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:38:54.351498 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:38:54.351515 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:38:54.352222 (+ 707us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:38:54.352621 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:38:54.352626 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:38:54.352630 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:38:54.352823 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:38:54.353370 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:38:54.353379 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:38:54.353726 (+ 347us) client_negotiation.cc:769] Sending connection context
0924 11:38:54.354019 (+ 293us) client_negotiation.cc:240] Negotiation successful
0924 11:38:54.354567 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":396,"threads_started":1}
I20260924 11:38:54.357561 25473 heartbeater.cc:347] Connected to a master server at 127.6.212.62:43561
I20260924 11:38:54.357887 25473 heartbeater.cc:464] Registering TS with master...
I20260924 11:38:54.358623 25473 heartbeater.cc:511] Master 127.6.212.62:43561 requested a full tablet report, sending...
I20260924 11:38:54.361258 25284 ts_manager.cc:194] Registered new tserver with Master: 2ffc01d8c4714548b1627fc67fc60edb (127.6.212.1:45239)
I20260924 11:38:54.363497 25284 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.1@KRBTEST.COM'} at 127.6.212.1:48893
WARNING: no policy specified for kudu/127.6.212.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab.
Entry for principal kudu/127.6.212.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab.
I20260924 11:38:54.400166 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.2:0
--local_ip_for_outbound_sockets=127.6.212.2
--webserver_interface=127.6.212.2
--webserver_port=0
--tserver_master_addrs=127.6.212.62:43561
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab
--principal=kudu/127.6.212.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:41023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:38:54.740899 25479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:38:54.741185 25479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:38:54.741382 25479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:38:54.749420 25479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:38:54.749543 25479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:38:54.749748 25479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.2
I20260924 11:38:54.760301 25479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:41023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab
--principal=kudu/127.6.212.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.212.2
--webserver_port=0
--tserver_master_addrs=127.6.212.62:43561
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.2
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:38:54.761677 25479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:38:54.763008 25479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:38:54.776976 25484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:38:54.776930 25485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:38:54.777014 25487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:38:54.778430 25479 server_base.cc:1034] running on GCE node
I20260924 11:38:54.779150 25479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:38:54.780227 25479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:38:54.781474 25479 hybrid_clock.cc:648] HybridClock initialized: now 1790249934781442 us; error 61 us; skew 500 ppm
I20260924 11:38:54.781807 25479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:38:54 dist-test-slave-cq7c krb5kdc[25239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249934, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:38:54.785459 25479 init.cc:377] Logged in from keytab as kudu/127.6.212.2@KRBTEST.COM (short username kudu)
I20260924 11:38:54.787204 25479 webserver.cc:460] Webserver started at http://127.6.212.2:42347/ using document root <none> and password file <none>
I20260924 11:38:54.788034 25479 fs_manager.cc:359] Metadata directory not provided
I20260924 11:38:54.788185 25479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:38:54.788506 25479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:38:54.791579 25479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-1/data/instance:
uuid: "0402889e34774af392532c47ba89e878"
format_stamp: "Formatted at 2026-09-24 11:38:54 on dist-test-slave-cq7c"
server_key: "5388ccdf65f1ca7eb991ee992a8e9b7b"
server_key_iv: "a5d58244f45ef3256c1744ada7c47d8f"
server_key_version: "encryptionkey@0"
I20260924 11:38:54.792418 25479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance:
uuid: "0402889e34774af392532c47ba89e878"
format_stamp: "Formatted at 2026-09-24 11:38:54 on dist-test-slave-cq7c"
server_key: "5388ccdf65f1ca7eb991ee992a8e9b7b"
server_key_iv: "a5d58244f45ef3256c1744ada7c47d8f"
server_key_version: "encryptionkey@0"
I20260924 11:38:54.797761 25479 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 11:38:54.801988 25494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:54.803733 25479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:38:54.803982 25479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-1/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-1/wal
uuid: "0402889e34774af392532c47ba89e878"
format_stamp: "Formatted at 2026-09-24 11:38:54 on dist-test-slave-cq7c"
server_key: "5388ccdf65f1ca7eb991ee992a8e9b7b"
server_key_iv: "a5d58244f45ef3256c1744ada7c47d8f"
server_key_version: "encryptionkey@0"
I20260924 11:38:54.804221 25479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:38:54.827579 25479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:38:54.832273 25479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:38:54.832592 25479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:38:54.833684 25479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:38:54.835079 25479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:38:54.835191 25479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:54.835347 25479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:38:54.835438 25479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260924 11:38:54.871488 25479 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.2:40097
I20260924 11:38:54.871575 25606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.2:40097 every 8 connection(s)
I20260924 11:38:54.872855 25479 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
I20260924 11:38:54.877734 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 25479
I20260924 11:38:54.877929 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance
I20260924 11:38:54.878398 6992 external_mini_cluster.cc:1428] Setting key 79a2e6f54fdbe05493bbc4b300a4b151
Sep 24 11:38:54 dist-test-slave-cq7c krb5kdc[25239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249934, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.2@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:38:54.896279 25609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:54.875702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:49607 (local address 127.6.212.62:43561)
0924 11:38:54.876151 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:54.876156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:54.877120 (+ 964us) server_negotiation.cc:407] Connection header received
0924 11:38:54.879196 (+ 2076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:54.879201 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:54.879297 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:54.879460 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:38:54.881430 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:54.882146 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:54.884849 (+ 2703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:54.885335 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:54.885441 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:54.890015 (+ 4574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:38:54.890020 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:38:54.890024 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:38:54.890059 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:38:54.892950 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:54.893717 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:54.893723 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:54.893732 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:54.893862 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:54.894321 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:54.894326 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:54.894333 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:54.894604 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:38:54.894780 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:54.895613 (+ 833us) server_negotiation.cc:299] Negotiation successful
0924 11:38:54.895866 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":232,"threads_started":1}
I20260924 11:38:54.897377 25610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:54.876246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:43561 (local address 127.6.212.2:49607)
0924 11:38:54.876937 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:54.876989 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 11:38:54.878914 (+ 1925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:38:54.879682 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:38:54.879691 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:38:54.880367 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:38:54.881226 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:54.881264 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:54.882309 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:54.882317 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:54.884689 (+ 2372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:54.884707 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:54.885586 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:54.885643 (+ 57us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:54.885998 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:54.886795 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:38:54.886831 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:38:54.889706 (+ 2875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:38:54.893095 (+ 3389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:38:54.893104 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:38:54.893122 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:38:54.893578 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:38:54.893992 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:38:54.893996 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:38:54.894000 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:38:54.894197 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:38:54.894759 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:38:54.894767 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:38:54.895148 (+ 381us) client_negotiation.cc:769] Sending connection context
0924 11:38:54.895518 (+ 370us) client_negotiation.cc:240] Negotiation successful
0924 11:38:54.896100 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":266,"threads_started":1}
I20260924 11:38:54.899036 25607 heartbeater.cc:347] Connected to a master server at 127.6.212.62:43561
I20260924 11:38:54.899443 25607 heartbeater.cc:464] Registering TS with master...
I20260924 11:38:54.900381 25607 heartbeater.cc:511] Master 127.6.212.62:43561 requested a full tablet report, sending...
I20260924 11:38:54.902200 25284 ts_manager.cc:194] Registered new tserver with Master: 0402889e34774af392532c47ba89e878 (127.6.212.2:40097)
I20260924 11:38:54.903574 25284 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.2@KRBTEST.COM'} at 127.6.212.2:49607
WARNING: no policy specified for kudu/127.6.212.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab.
Entry for principal kudu/127.6.212.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab.
I20260924 11:38:54.942912 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.3:0
--local_ip_for_outbound_sockets=127.6.212.3
--webserver_interface=127.6.212.3
--webserver_port=0
--tserver_master_addrs=127.6.212.62:43561
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab
--principal=kudu/127.6.212.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:41023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:38:55.285323 25613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:38:55.285624 25613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:38:55.285771 25613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:38:55.293468 25613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:38:55.293597 25613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:38:55.293821 25613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.3
I20260924 11:38:55.305007 25613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:41023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab
--principal=kudu/127.6.212.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.212.3
--webserver_port=0
--tserver_master_addrs=127.6.212.62:43561
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.3
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:38:55.306417 25613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:38:55.307873 25613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:38:55.322444 25619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:38:55.322420 25621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:38:55.322736 25618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:38:55.323223 25613 server_base.cc:1034] running on GCE node
I20260924 11:38:55.323922 25613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:38:55.325042 25613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:38:55.326288 25613 hybrid_clock.cc:648] HybridClock initialized: now 1790249935326222 us; error 60 us; skew 500 ppm
I20260924 11:38:55.326655 25613 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:38:55 dist-test-slave-cq7c krb5kdc[25239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249935, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:38:55.330057 25613 init.cc:377] Logged in from keytab as kudu/127.6.212.3@KRBTEST.COM (short username kudu)
I20260924 11:38:55.332118 25613 webserver.cc:460] Webserver started at http://127.6.212.3:43169/ using document root <none> and password file <none>
I20260924 11:38:55.333029 25613 fs_manager.cc:359] Metadata directory not provided
I20260924 11:38:55.333168 25613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:38:55.333534 25613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:38:55.337580 25613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-2/data/instance:
uuid: "a65fe648a2d34d89a127503458b24088"
format_stamp: "Formatted at 2026-09-24 11:38:55 on dist-test-slave-cq7c"
server_key: "18208a702c81857d868f4f4be0c29fdf"
server_key_iv: "7f9a818743995e3827271d6031ce7079"
server_key_version: "encryptionkey@0"
I20260924 11:38:55.338321 25613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance:
uuid: "a65fe648a2d34d89a127503458b24088"
format_stamp: "Formatted at 2026-09-24 11:38:55 on dist-test-slave-cq7c"
server_key: "18208a702c81857d868f4f4be0c29fdf"
server_key_iv: "7f9a818743995e3827271d6031ce7079"
server_key_version: "encryptionkey@0"
I20260924 11:38:55.343871 25613 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260924 11:38:55.347816 25628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:55.349326 25613 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:38:55.349531 25613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-2/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-2/wal
uuid: "a65fe648a2d34d89a127503458b24088"
format_stamp: "Formatted at 2026-09-24 11:38:55 on dist-test-slave-cq7c"
server_key: "18208a702c81857d868f4f4be0c29fdf"
server_key_iv: "7f9a818743995e3827271d6031ce7079"
server_key_version: "encryptionkey@0"
I20260924 11:38:55.349771 25613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:38:55.365473 25613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:38:55.366950 25473 heartbeater.cc:503] Master 127.6.212.62:43561 was elected leader, sending a full tablet report...
I20260924 11:38:55.370743 25613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:38:55.371016 25613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:38:55.372177 25613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:38:55.373545 25613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:38:55.373616 25613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:55.373699 25613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:38:55.373749 25613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:55.409364 25613 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.3:45371
I20260924 11:38:55.409461 25740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.3:45371 every 8 connection(s)
I20260924 11:38:55.410890 25613 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
I20260924 11:38:55.420390 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 25613
I20260924 11:38:55.420586 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance
I20260924 11:38:55.421018 6992 external_mini_cluster.cc:1428] Setting key 320aa05a06abaf57aca56561cae8b5f5
Sep 24 11:38:55 dist-test-slave-cq7c krb5kdc[25239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249935, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.3@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:38:55.433637 25743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:55.413663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:57469 (local address 127.6.212.62:43561)
0924 11:38:55.414196 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:55.414202 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:55.415212 (+ 1010us) server_negotiation.cc:407] Connection header received
0924 11:38:55.417269 (+ 2057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:55.417274 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:55.417356 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:55.417466 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:38:55.419166 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:55.420247 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:55.422398 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:55.422840 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:55.423040 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:55.427035 (+ 3995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:38:55.427041 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:38:55.427044 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:38:55.427082 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:38:55.429990 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:55.430895 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:55.430899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:55.430901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:55.431049 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:55.431610 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:55.431617 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:55.431628 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:55.431944 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:38:55.432136 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:55.432936 (+ 800us) server_negotiation.cc:299] Negotiation successful
0924 11:38:55.433193 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":263,"threads_started":1}
I20260924 11:38:55.434614 25744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:55.414203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:43561 (local address 127.6.212.3:57469)
0924 11:38:55.415029 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:55.415071 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 11:38:55.416976 (+ 1905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:38:55.417723 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:38:55.417732 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:38:55.418294 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:38:55.418943 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:55.418961 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:55.420453 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:55.420484 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:55.422138 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:55.422188 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:55.423130 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:55.423140 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:55.423541 (+ 401us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:55.424375 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:38:55.424401 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:38:55.426778 (+ 2377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:38:55.430165 (+ 3387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:38:55.430176 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:38:55.430199 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:38:55.430657 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:38:55.431189 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:38:55.431200 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:38:55.431203 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:38:55.431485 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:38:55.432088 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:38:55.432100 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:38:55.432489 (+ 389us) client_negotiation.cc:769] Sending connection context
0924 11:38:55.432805 (+ 316us) client_negotiation.cc:240] Negotiation successful
0924 11:38:55.433396 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":327,"threads_started":1}
I20260924 11:38:55.436232 25741 heartbeater.cc:347] Connected to a master server at 127.6.212.62:43561
I20260924 11:38:55.436601 25741 heartbeater.cc:464] Registering TS with master...
I20260924 11:38:55.437351 25741 heartbeater.cc:511] Master 127.6.212.62:43561 requested a full tablet report, sending...
I20260924 11:38:55.438882 25284 ts_manager.cc:194] Registered new tserver with Master: a65fe648a2d34d89a127503458b24088 (127.6.212.3:45371)
I20260924 11:38:55.439805 25284 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.3@KRBTEST.COM'} at 127.6.212.3:57469
I20260924 11:38:55.449330 6992 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:38:55.467646 25743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:55.453869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41682 (local address 127.6.212.62:43561)
0924 11:38:55.454115 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:55.454122 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:55.454450 (+ 328us) server_negotiation.cc:407] Connection header received
0924 11:38:55.454655 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:55.454660 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:55.454731 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:55.454869 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:38:55.456517 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:55.457276 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:55.458638 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:55.459059 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:55.459159 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:55.462031 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:38:55.462036 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:38:55.462039 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:38:55.462072 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:38:55.464812 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:55.465600 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:55.465604 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:55.465606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:55.465719 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:55.466213 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:55.466217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:55.466219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:55.466545 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:38:55.466739 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:55.467160 (+ 421us) server_negotiation.cc:299] Negotiation successful
0924 11:38:55.467309 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 11:38:55.471925 25284 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41682:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:38:55.479108 25284 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:38:55.505395 25754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:55.492632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.3:45371 (local address 127.0.0.1:39686)
0924 11:38:55.495011 (+ 2379us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:55.495055 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 11:38:55.495228 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:38:55.496928 (+ 1700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:38:55.496936 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:38:55.496968 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:38:55.497446 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:55.497465 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:55.499981 (+ 2516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:55.499986 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:55.502625 (+ 2639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:55.502637 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:55.504157 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:55.504164 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:55.504450 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:55.504512 (+ 62us) client_negotiation.cc:769] Sending connection context
0924 11:38:55.504674 (+ 162us) client_negotiation.cc:240] Negotiation successful
0924 11:38:55.504895 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2136,"mutex_wait_us":5,"thread_start_us":2019,"threads_started":1}
I20260924 11:38:55.506367 25755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:55.493052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39686 (local address 127.6.212.3:45371)
0924 11:38:55.494060 (+ 1008us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:55.494067 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:55.495180 (+ 1113us) server_negotiation.cc:407] Connection header received
0924 11:38:55.495503 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:55.495511 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:55.495829 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:55.496076 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:38:55.497942 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:55.499769 (+ 1827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:55.502800 (+ 3031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:55.503922 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:55.504093 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:55.504372 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:55.505882 (+ 1510us) server_negotiation.cc:299] Negotiation successful
0924 11:38:55.506024 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":851,"thread_start_us":656,"threads_started":1}
I20260924 11:38:55.507400 25756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:55.495501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.2:40097 (local address 127.0.0.1:41846)
0924 11:38:55.496503 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:55.496522 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:38:55.496637 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:38:55.499438 (+ 2801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:38:55.499443 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:38:55.499470 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:38:55.499987 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:55.499998 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:55.502166 (+ 2168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:55.502170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:55.504432 (+ 2262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:55.504446 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:55.506356 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:55.506366 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:55.506698 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:55.506739 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 11:38:55.506861 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 11:38:55.507016 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":874,"thread_start_us":727,"threads_started":1}
I20260924 11:38:55.507519 25757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:55.495800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41846 (local address 127.6.212.2:40097)
0924 11:38:55.498784 (+ 2984us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:55.498798 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:55.498820 (+ 22us) server_negotiation.cc:407] Connection header received
0924 11:38:55.498942 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:55.498950 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:55.499203 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:55.499425 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:38:55.500213 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:55.502026 (+ 1813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:55.504621 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:55.506113 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:55.506252 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:55.506461 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:55.506920 (+ 459us) server_negotiation.cc:299] Negotiation successful
0924 11:38:55.507100 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2831,"thread_start_us":2698,"threads_started":1}
I20260924 11:38:55.507865 25758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:55.497059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34596 (local address 127.6.212.1:45239)
0924 11:38:55.497821 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:55.497830 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:55.498683 (+ 853us) server_negotiation.cc:407] Connection header received
0924 11:38:55.498982 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:55.498989 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:55.499200 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:55.499424 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:38:55.500509 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:55.501760 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:55.504938 (+ 3178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:55.506429 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:55.506529 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:55.506738 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:55.507298 (+ 560us) server_negotiation.cc:299] Negotiation successful
0924 11:38:55.507498 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":388,"threads_started":1}
I20260924 11:38:55.508570 25759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:55.497846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:45239 (local address 127.0.0.1:34596)
0924 11:38:55.498564 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:55.498583 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:38:55.498785 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:38:55.499714 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:38:55.499719 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:38:55.499753 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:38:55.500280 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:55.500300 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:55.501905 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:55.501912 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:55.504756 (+ 2844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:55.504768 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:55.506942 (+ 2174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:55.506948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:55.507105 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:55.507141 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 11:38:55.508070 (+ 929us) client_negotiation.cc:240] Negotiation successful
0924 11:38:55.508210 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"mutex_wait_us":95,"thread_start_us":361,"threads_started":1}
I20260924 11:38:55.510984 25676 tablet_service.cc:1467] Processing CreateTablet for tablet 14d2710bf01943fc90c13d10100e6e89 (DEFAULT_TABLE table=test-table [id=2b1f29c7a7b84a03978ed20750737a61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:38:55.512609 25676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14d2710bf01943fc90c13d10100e6e89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:38:55.512589 25408 tablet_service.cc:1467] Processing CreateTablet for tablet 14d2710bf01943fc90c13d10100e6e89 (DEFAULT_TABLE table=test-table [id=2b1f29c7a7b84a03978ed20750737a61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:38:55.513993 25408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14d2710bf01943fc90c13d10100e6e89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:38:55.513682 25542 tablet_service.cc:1467] Processing CreateTablet for tablet 14d2710bf01943fc90c13d10100e6e89 (DEFAULT_TABLE table=test-table [id=2b1f29c7a7b84a03978ed20750737a61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:38:55.514809 25542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14d2710bf01943fc90c13d10100e6e89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:38:55.528160 25760 tablet_bootstrap.cc:492] T 14d2710bf01943fc90c13d10100e6e89 P a65fe648a2d34d89a127503458b24088: Bootstrap starting.
I20260924 11:38:55.530745 25761 tablet_bootstrap.cc:492] T 14d2710bf01943fc90c13d10100e6e89 P 0402889e34774af392532c47ba89e878: Bootstrap starting.
I20260924 11:38:55.531663 25762 tablet_bootstrap.cc:492] T 14d2710bf01943fc90c13d10100e6e89 P 2ffc01d8c4714548b1627fc67fc60edb: Bootstrap starting.
I20260924 11:38:55.533783 25760 tablet_bootstrap.cc:654] T 14d2710bf01943fc90c13d10100e6e89 P a65fe648a2d34d89a127503458b24088: Neither blocks nor log segments found. Creating new log.
I20260924 11:38:55.534642 25761 tablet_bootstrap.cc:654] T 14d2710bf01943fc90c13d10100e6e89 P 0402889e34774af392532c47ba89e878: Neither blocks nor log segments found. Creating new log.
I20260924 11:38:55.535151 25760 log.cc:824] T 14d2710bf01943fc90c13d10100e6e89 P a65fe648a2d34d89a127503458b24088: Log is configured to *not* fsync() on all Append() calls
I20260924 11:38:55.535535 25762 tablet_bootstrap.cc:654] T 14d2710bf01943fc90c13d10100e6e89 P 2ffc01d8c4714548b1627fc67fc60edb: Neither blocks nor log segments found. Creating new log.
I20260924 11:38:55.535964 25761 log.cc:824] T 14d2710bf01943fc90c13d10100e6e89 P 0402889e34774af392532c47ba89e878: Log is configured to *not* fsync() on all Append() calls
I20260924 11:38:55.536921 25762 log.cc:824] T 14d2710bf01943fc90c13d10100e6e89 P 2ffc01d8c4714548b1627fc67fc60edb: Log is configured to *not* fsync() on all Append() calls
I20260924 11:38:55.538697 25760 tablet_bootstrap.cc:492] T 14d2710bf01943fc90c13d10100e6e89 P a65fe648a2d34d89a127503458b24088: No bootstrap required, opened a new log
I20260924 11:38:55.538874 25761 tablet_bootstrap.cc:492] T 14d2710bf01943fc90c13d10100e6e89 P 0402889e34774af392532c47ba89e878: No bootstrap required, opened a new log
I20260924 11:38:55.539011 25760 ts_tablet_manager.cc:1397] T 14d2710bf01943fc90c13d10100e6e89 P a65fe648a2d34d89a127503458b24088: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.005s
I20260924 11:38:55.539172 25761 ts_tablet_manager.cc:1397] T 14d2710bf01943fc90c13d10100e6e89 P 0402889e34774af392532c47ba89e878: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260924 11:38:55.540072 25762 tablet_bootstrap.cc:492] T 14d2710bf01943fc90c13d10100e6e89 P 2ffc01d8c4714548b1627fc67fc60edb: No bootstrap required, opened a new log
I20260924 11:38:55.540350 25762 ts_tablet_manager.cc:1397] T 14d2710bf01943fc90c13d10100e6e89 P 2ffc01d8c4714548b1627fc67fc60edb: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260924 11:38:55.547274 25761 raft_consensus.cc:348] T 14d2710bf01943fc90c13d10100e6e89 P 0402889e34774af392532c47ba89e878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a65fe648a2d34d89a127503458b24088" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 45371 } } peers { permanent_uuid: "0402889e34774af392532c47ba89e878" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 40097 } } peers { permanent_uuid: "2ffc01d8c4714548b1627fc67fc60edb" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 45239 } }
I20260924 11:38:55.547770 25761 raft_consensus.cc:374] T 14d2710bf01943fc90c13d10100e6e89 P 0402889e34774af392532c47ba89e878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:38:55.547865 25761 raft_consensus.cc:729] T 14d2710bf01943fc90c13d10100e6e89 P 0402889e34774af392532c47ba89e878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0402889e34774af392532c47ba89e878, State: Initialized, Role: FOLLOWER
I20260924 11:38:55.548530 25761 consensus_queue.cc:260] T 14d2710bf01943fc90c13d10100e6e89 P 0402889e34774af392532c47ba89e878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a65fe648a2d34d89a127503458b24088" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 45371 } } peers { permanent_uuid: "0402889e34774af392532c47ba89e878" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 40097 } } peers { permanent_uuid: "2ffc01d8c4714548b1627fc67fc60edb" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 45239 } }
I20260924 11:38:55.551433 25607 heartbeater.cc:503] Master 127.6.212.62:43561 was elected leader, sending a full tablet report...
I20260924 11:38:55.551172 25760 raft_consensus.cc:348] T 14d2710bf01943fc90c13d10100e6e89 P a65fe648a2d34d89a127503458b24088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a65fe648a2d34d89a127503458b24088" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 45371 } } peers { permanent_uuid: "0402889e34774af392532c47ba89e878" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 40097 } } peers { permanent_uuid: "2ffc01d8c4714548b1627fc67fc60edb" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 45239 } }
I20260924 11:38:55.551719 25760 raft_consensus.cc:374] T 14d2710bf01943fc90c13d10100e6e89 P a65fe648a2d34d89a127503458b24088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:38:55.551878 25760 raft_consensus.cc:729] T 14d2710bf01943fc90c13d10100e6e89 P a65fe648a2d34d89a127503458b24088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a65fe648a2d34d89a127503458b24088, State: Initialized, Role: FOLLOWER
I20260924 11:38:55.552089 25761 ts_tablet_manager.cc:1428] T 14d2710bf01943fc90c13d10100e6e89 P 0402889e34774af392532c47ba89e878: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260924 11:38:55.552613 25760 consensus_queue.cc:260] T 14d2710bf01943fc90c13d10100e6e89 P a65fe648a2d34d89a127503458b24088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a65fe648a2d34d89a127503458b24088" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 45371 } } peers { permanent_uuid: "0402889e34774af392532c47ba89e878" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 40097 } } peers { permanent_uuid: "2ffc01d8c4714548b1627fc67fc60edb" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 45239 } }
I20260924 11:38:55.553826 25741 heartbeater.cc:503] Master 127.6.212.62:43561 was elected leader, sending a full tablet report...
I20260924 11:38:55.554589 25760 ts_tablet_manager.cc:1428] T 14d2710bf01943fc90c13d10100e6e89 P a65fe648a2d34d89a127503458b24088: Time spent starting tablet: real 0.015s user 0.008s sys 0.008s
I20260924 11:38:55.556097 25762 raft_consensus.cc:348] T 14d2710bf01943fc90c13d10100e6e89 P 2ffc01d8c4714548b1627fc67fc60edb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a65fe648a2d34d89a127503458b24088" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 45371 } } peers { permanent_uuid: "0402889e34774af392532c47ba89e878" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 40097 } } peers { permanent_uuid: "2ffc01d8c4714548b1627fc67fc60edb" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 45239 } }
I20260924 11:38:55.556656 25762 raft_consensus.cc:374] T 14d2710bf01943fc90c13d10100e6e89 P 2ffc01d8c4714548b1627fc67fc60edb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:38:55.556773 25762 raft_consensus.cc:729] T 14d2710bf01943fc90c13d10100e6e89 P 2ffc01d8c4714548b1627fc67fc60edb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ffc01d8c4714548b1627fc67fc60edb, State: Initialized, Role: FOLLOWER
I20260924 11:38:55.557489 25762 consensus_queue.cc:260] T 14d2710bf01943fc90c13d10100e6e89 P 2ffc01d8c4714548b1627fc67fc60edb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a65fe648a2d34d89a127503458b24088" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 45371 } } peers { permanent_uuid: "0402889e34774af392532c47ba89e878" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 40097 } } peers { permanent_uuid: "2ffc01d8c4714548b1627fc67fc60edb" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 45239 } }
I20260924 11:38:55.561357 25762 ts_tablet_manager.cc:1428] T 14d2710bf01943fc90c13d10100e6e89 P 2ffc01d8c4714548b1627fc67fc60edb: Time spent starting tablet: real 0.021s user 0.017s sys 0.000s
I20260924 11:38:55.582916 25766 raft_consensus.cc:482] T 14d2710bf01943fc90c13d10100e6e89 P 0402889e34774af392532c47ba89e878 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:38:55.583271 25766 raft_consensus.cc:504] T 14d2710bf01943fc90c13d10100e6e89 P 0402889e34774af392532c47ba89e878 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a65fe648a2d34d89a127503458b24088" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 45371 } } peers { permanent_uuid: "0402889e34774af392532c47ba89e878" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 40097 } } peers { permanent_uuid: "2ffc01d8c4714548b1627fc67fc60edb" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 45239 } }
W20260924 11:38:55.584760 25474 tablet.cc:2369] T 14d2710bf01943fc90c13d10100e6e89 P 2ffc01d8c4714548b1627fc67fc60edb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:38:55.585206 25766 leader_election.cc:288] T 14d2710bf01943fc90c13d10100e6e89 P 0402889e34774af392532c47ba89e878 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a65fe648a2d34d89a127503458b24088 (127.6.212.3:45371), 2ffc01d8c4714548b1627fc67fc60edb (127.6.212.1:45239)
I20260924 11:38:55.592022 25755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:55.585670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:37191 (local address 127.6.212.3:45371)
0924 11:38:55.585862 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:55.585867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:55.586545 (+ 678us) server_negotiation.cc:407] Connection header received
0924 11:38:55.586760 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:55.586765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:55.586848 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:55.586960 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:38:55.587699 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:55.588453 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:55.590227 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:55.590991 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:55.591144 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:55.591270 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:55.591613 (+ 343us) server_negotiation.cc:299] Negotiation successful
0924 11:38:55.591733 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 11:38:55.592262 25769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:55.585651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.3:45371 (local address 127.6.212.2:37191)
0924 11:38:55.586424 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:55.586449 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 11:38:55.586592 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:38:55.587086 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:38:55.587092 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:38:55.587120 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:38:55.587551 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:55.587560 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:55.588584 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:55.588592 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:55.590059 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:55.590070 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:55.591232 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:55.591241 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:55.591449 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:55.591488 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 11:38:55.591589 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 11:38:55.591734 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":413,"threads_started":1}
I20260924 11:38:55.592922 25770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:55.586530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:45239 (local address 127.6.212.2:55561)
0924 11:38:55.587162 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:55.587175 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 11:38:55.587337 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:38:55.587688 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:38:55.587692 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:38:55.587722 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:38:55.588030 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:55.588039 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:55.589039 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:55.589046 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:55.590424 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:55.590434 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:55.592026 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:55.592035 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:55.592363 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:55.592409 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 11:38:55.592509 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 11:38:55.592635 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":369,"threads_started":1}
I20260924 11:38:55.592918 25758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:55.586718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:55561 (local address 127.6.212.1:45239)
0924 11:38:55.586995 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:55.587001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:55.587268 (+ 267us) server_negotiation.cc:407] Connection header received
0924 11:38:55.587509 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:55.587518 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:55.587575 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:55.587676 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:38:55.588182 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:55.588900 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:55.590579 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:55.591645 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:55.591763 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:55.591856 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:55.592542 (+ 686us) server_negotiation.cc:299] Negotiation successful
0924 11:38:55.592665 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 11:38:55.593338 25696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14d2710bf01943fc90c13d10100e6e89" candidate_uuid: "0402889e34774af392532c47ba89e878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a65fe648a2d34d89a127503458b24088" is_pre_election: true
I20260924 11:38:55.593919 25696 raft_consensus.cc:2454] T 14d2710bf01943fc90c13d10100e6e89 P a65fe648a2d34d89a127503458b24088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0402889e34774af392532c47ba89e878 in term 0.
I20260924 11:38:55.594067 25428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14d2710bf01943fc90c13d10100e6e89" candidate_uuid: "0402889e34774af392532c47ba89e878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ffc01d8c4714548b1627fc67fc60edb" is_pre_election: true
I20260924 11:38:55.594609 25428 raft_consensus.cc:2454] T 14d2710bf01943fc90c13d10100e6e89 P 2ffc01d8c4714548b1627fc67fc60edb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0402889e34774af392532c47ba89e878 in term 0.
I20260924 11:38:55.594673 25498 leader_election.cc:302] T 14d2710bf01943fc90c13d10100e6e89 P 0402889e34774af392532c47ba89e878 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0402889e34774af392532c47ba89e878, a65fe648a2d34d89a127503458b24088; no voters:
I20260924 11:38:55.595099 25766 raft_consensus.cc:2781] T 14d2710bf01943fc90c13d10100e6e89 P 0402889e34774af392532c47ba89e878 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:38:55.595269 25766 raft_consensus.cc:482] T 14d2710bf01943fc90c13d10100e6e89 P 0402889e34774af392532c47ba89e878 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:38:55.595397 25766 raft_consensus.cc:3037] T 14d2710bf01943fc90c13d10100e6e89 P 0402889e34774af392532c47ba89e878 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:38:55.597828 25766 raft_consensus.cc:504] T 14d2710bf01943fc90c13d10100e6e89 P 0402889e34774af392532c47ba89e878 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a65fe648a2d34d89a127503458b24088" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 45371 } } peers { permanent_uuid: "0402889e34774af392532c47ba89e878" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 40097 } } peers { permanent_uuid: "2ffc01d8c4714548b1627fc67fc60edb" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 45239 } }
I20260924 11:38:55.598523 25766 leader_election.cc:288] T 14d2710bf01943fc90c13d10100e6e89 P 0402889e34774af392532c47ba89e878 [CANDIDATE]: Term 1 election: Requested vote from peers a65fe648a2d34d89a127503458b24088 (127.6.212.3:45371), 2ffc01d8c4714548b1627fc67fc60edb (127.6.212.1:45239)
I20260924 11:38:55.599092 25696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14d2710bf01943fc90c13d10100e6e89" candidate_uuid: "0402889e34774af392532c47ba89e878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a65fe648a2d34d89a127503458b24088"
I20260924 11:38:55.599182 25428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14d2710bf01943fc90c13d10100e6e89" candidate_uuid: "0402889e34774af392532c47ba89e878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ffc01d8c4714548b1627fc67fc60edb"
I20260924 11:38:55.599311 25696 raft_consensus.cc:3037] T 14d2710bf01943fc90c13d10100e6e89 P a65fe648a2d34d89a127503458b24088 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:38:55.599486 25428 raft_consensus.cc:3037] T 14d2710bf01943fc90c13d10100e6e89 P 2ffc01d8c4714548b1627fc67fc60edb [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:38:55.601742 25696 raft_consensus.cc:2454] T 14d2710bf01943fc90c13d10100e6e89 P a65fe648a2d34d89a127503458b24088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0402889e34774af392532c47ba89e878 in term 1.
I20260924 11:38:55.601820 25428 raft_consensus.cc:2454] T 14d2710bf01943fc90c13d10100e6e89 P 2ffc01d8c4714548b1627fc67fc60edb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0402889e34774af392532c47ba89e878 in term 1.
I20260924 11:38:55.602273 25498 leader_election.cc:302] T 14d2710bf01943fc90c13d10100e6e89 P 0402889e34774af392532c47ba89e878 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0402889e34774af392532c47ba89e878, a65fe648a2d34d89a127503458b24088; no voters:
I20260924 11:38:55.602620 25766 raft_consensus.cc:2781] T 14d2710bf01943fc90c13d10100e6e89 P 0402889e34774af392532c47ba89e878 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:38:55.603214 25766 raft_consensus.cc:686] T 14d2710bf01943fc90c13d10100e6e89 P 0402889e34774af392532c47ba89e878 [term 1 LEADER]: Becoming Leader. State: Replica: 0402889e34774af392532c47ba89e878, State: Running, Role: LEADER
I20260924 11:38:55.603856 25766 consensus_queue.cc:237] T 14d2710bf01943fc90c13d10100e6e89 P 0402889e34774af392532c47ba89e878 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a65fe648a2d34d89a127503458b24088" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 45371 } } peers { permanent_uuid: "0402889e34774af392532c47ba89e878" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 40097 } } peers { permanent_uuid: "2ffc01d8c4714548b1627fc67fc60edb" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 45239 } }
I20260924 11:38:55.608687 25282 catalog_manager.cc:5495] T 14d2710bf01943fc90c13d10100e6e89 P 0402889e34774af392532c47ba89e878 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0402889e34774af392532c47ba89e878 (127.6.212.2). New cstate: current_term: 1 leader_uuid: "0402889e34774af392532c47ba89e878" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a65fe648a2d34d89a127503458b24088" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 45371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0402889e34774af392532c47ba89e878" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 40097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ffc01d8c4714548b1627fc67fc60edb" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 45239 } health_report { overall_health: UNKNOWN } } }
W20260924 11:38:55.624878 25608 tablet.cc:2369] T 14d2710bf01943fc90c13d10100e6e89 P 0402889e34774af392532c47ba89e878: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:38:55.632488 25757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:55.627026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41858 (local address 127.6.212.2:40097)
0924 11:38:55.627206 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:55.627210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:55.627228 (+ 18us) server_negotiation.cc:407] Connection header received
0924 11:38:55.627457 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:55.627461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:55.627547 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:55.627677 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:38:55.628078 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:55.628801 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:55.630216 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:55.630648 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:55.630749 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:55.631297 (+ 548us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:38:55.631922 (+ 625us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:38:55.632056 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:55.632200 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 11:38:55.632241 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 11:38:55.641878 25428 raft_consensus.cc:1261] T 14d2710bf01943fc90c13d10100e6e89 P 2ffc01d8c4714548b1627fc67fc60edb [term 1 FOLLOWER]: Refusing update from remote peer 0402889e34774af392532c47ba89e878: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:38:55.641870 25696 raft_consensus.cc:1261] T 14d2710bf01943fc90c13d10100e6e89 P a65fe648a2d34d89a127503458b24088 [term 1 FOLLOWER]: Refusing update from remote peer 0402889e34774af392532c47ba89e878: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:38:55.642958 25766 consensus_queue.cc:1035] T 14d2710bf01943fc90c13d10100e6e89 P 0402889e34774af392532c47ba89e878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a65fe648a2d34d89a127503458b24088" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 45371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:38:55.643321 25771 consensus_queue.cc:1035] T 14d2710bf01943fc90c13d10100e6e89 P 0402889e34774af392532c47ba89e878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ffc01d8c4714548b1627fc67fc60edb" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 45239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:38:55.658346 25774 mvcc.cc:205] Tried to move back new op lower bound from 7332863736377335808 to 7332863736237715456. Current Snapshot: MvccSnapshot[applied={T|T < 7332863736377335808}]
I20260924 11:38:55.679062 25282 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:38:55.679414 25282 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:38:55.683646 25282 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ab5b92dab57e4c148f77c2481a280c07: Sending DeleteTablet for 3 replicas of tablet 14d2710bf01943fc90c13d10100e6e89
I20260924 11:38:55.685011 25676 tablet_service.cc:1514] Processing DeleteTablet for tablet 14d2710bf01943fc90c13d10100e6e89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:38:55 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:39686
I20260924 11:38:55.685137 25408 tablet_service.cc:1514] Processing DeleteTablet for tablet 14d2710bf01943fc90c13d10100e6e89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:38:55 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:34596
I20260924 11:38:55.685194 25542 tablet_service.cc:1514] Processing DeleteTablet for tablet 14d2710bf01943fc90c13d10100e6e89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:38:55 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:41846
I20260924 11:38:55.690331 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 25345
I20260924 11:38:55.700637 25786 tablet_replica.cc:330] T 14d2710bf01943fc90c13d10100e6e89 P 0402889e34774af392532c47ba89e878: stopping tablet replica
I20260924 11:38:55.701277 25786 raft_consensus.cc:2229] T 14d2710bf01943fc90c13d10100e6e89 P 0402889e34774af392532c47ba89e878 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:38:55.702286 25786 raft_consensus.cc:2258] T 14d2710bf01943fc90c13d10100e6e89 P 0402889e34774af392532c47ba89e878 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:38:55.705159 25786 ts_tablet_manager.cc:1905] T 14d2710bf01943fc90c13d10100e6e89 P 0402889e34774af392532c47ba89e878: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:38:55.718389 25786 ts_tablet_manager.cc:1918] T 14d2710bf01943fc90c13d10100e6e89 P 0402889e34774af392532c47ba89e878: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:38:55.718613 25786 log.cc:1196] T 14d2710bf01943fc90c13d10100e6e89 P 0402889e34774af392532c47ba89e878: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-1/wal/wals/14d2710bf01943fc90c13d10100e6e89
I20260924 11:38:55.719264 25786 ts_tablet_manager.cc:1939] T 14d2710bf01943fc90c13d10100e6e89 P 0402889e34774af392532c47ba89e878: Deleting consensus metadata
I20260924 11:38:55.721587 25271 catalog_manager.cc:4841] TS 0402889e34774af392532c47ba89e878 (127.6.212.2:40097): tablet 14d2710bf01943fc90c13d10100e6e89 (table test-table [id=2b1f29c7a7b84a03978ed20750737a61]) successfully deleted
I20260924 11:38:55.722487 25784 tablet_replica.cc:330] T 14d2710bf01943fc90c13d10100e6e89 P a65fe648a2d34d89a127503458b24088: stopping tablet replica
I20260924 11:38:55.722793 25785 tablet_replica.cc:330] T 14d2710bf01943fc90c13d10100e6e89 P 2ffc01d8c4714548b1627fc67fc60edb: stopping tablet replica
I20260924 11:38:55.723006 25784 raft_consensus.cc:2229] T 14d2710bf01943fc90c13d10100e6e89 P a65fe648a2d34d89a127503458b24088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:38:55.723387 25785 raft_consensus.cc:2229] T 14d2710bf01943fc90c13d10100e6e89 P 2ffc01d8c4714548b1627fc67fc60edb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:38:55.723662 25784 raft_consensus.cc:2258] T 14d2710bf01943fc90c13d10100e6e89 P a65fe648a2d34d89a127503458b24088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:38:55.724027 25785 raft_consensus.cc:2258] T 14d2710bf01943fc90c13d10100e6e89 P 2ffc01d8c4714548b1627fc67fc60edb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:38:55.726087 25784 ts_tablet_manager.cc:1905] T 14d2710bf01943fc90c13d10100e6e89 P a65fe648a2d34d89a127503458b24088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:38:55.726832 25785 ts_tablet_manager.cc:1905] T 14d2710bf01943fc90c13d10100e6e89 P 2ffc01d8c4714548b1627fc67fc60edb: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 11:38:55 dist-test-slave-cq7c krb5kdc[25239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.1@KRBTEST.COM
I20260924 11:38:55.733676 25784 ts_tablet_manager.cc:1918] T 14d2710bf01943fc90c13d10100e6e89 P a65fe648a2d34d89a127503458b24088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:38:55.733882 25784 log.cc:1196] T 14d2710bf01943fc90c13d10100e6e89 P a65fe648a2d34d89a127503458b24088: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-2/wal/wals/14d2710bf01943fc90c13d10100e6e89
I20260924 11:38:55.734436 25784 ts_tablet_manager.cc:1939] T 14d2710bf01943fc90c13d10100e6e89 P a65fe648a2d34d89a127503458b24088: Deleting consensus metadata
I20260924 11:38:55.735415 25785 ts_tablet_manager.cc:1918] T 14d2710bf01943fc90c13d10100e6e89 P 2ffc01d8c4714548b1627fc67fc60edb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:38:55.735641 25785 log.cc:1196] T 14d2710bf01943fc90c13d10100e6e89 P 2ffc01d8c4714548b1627fc67fc60edb: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790249617057452-6992-0/minicluster-data/ts-0/wal/wals/14d2710bf01943fc90c13d10100e6e89
I20260924 11:38:55.736228 25785 ts_tablet_manager.cc:1939] T 14d2710bf01943fc90c13d10100e6e89 P 2ffc01d8c4714548b1627fc67fc60edb: Deleting consensus metadata
I20260924 11:38:55.736549 25271 catalog_manager.cc:4841] TS a65fe648a2d34d89a127503458b24088 (127.6.212.3:45371): tablet 14d2710bf01943fc90c13d10100e6e89 (table test-table [id=2b1f29c7a7b84a03978ed20750737a61]) successfully deleted
I20260924 11:38:55.738232 25271 catalog_manager.cc:4841] TS 2ffc01d8c4714548b1627fc67fc60edb (127.6.212.1:45239): tablet 14d2710bf01943fc90c13d10100e6e89 (table test-table [id=2b1f29c7a7b84a03978ed20750737a61]) successfully deleted
I20260924 11:38:55.738667 25758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:55.692189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34602 (local address 127.6.212.1:45239)
0924 11:38:55.692444 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:55.692453 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:55.717771 (+ 25318us) server_negotiation.cc:407] Connection header received
0924 11:38:55.718341 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:55.718350 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:55.718484 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:55.718571 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:38:55.720569 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:55.721680 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:55.723928 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:55.724669 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:55.724775 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:55.730690 (+ 5915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:38:55.730707 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:38:55.730728 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:38:55.730776 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:38:55.735065 (+ 4289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:55.736129 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:55.736139 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:55.736142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:55.736342 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:55.736874 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:55.736879 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:55.736883 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:55.737275 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:38:55.737508 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:55.738047 (+ 539us) server_negotiation.cc:299] Negotiation successful
0924 11:38:55.738273 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 11:38:55.739404 25468 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:38:55.832916 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 25345
I20260924 11:38:55.851424 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 25479
Sep 24 11:38:55 dist-test-slave-cq7c krb5kdc[25239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.2@KRBTEST.COM
I20260924 11:38:55.870711 25757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:55.852272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41860 (local address 127.6.212.2:40097)
0924 11:38:55.852462 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:55.852467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:55.852523 (+ 56us) server_negotiation.cc:407] Connection header received
0924 11:38:55.852740 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:55.852744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:55.852825 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:55.852916 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:38:55.854759 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:55.855937 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:55.857904 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:55.858359 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:55.858461 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:55.864012 (+ 5551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:38:55.864029 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:38:55.864046 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:38:55.864103 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:38:55.867328 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:55.868272 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:55.868281 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:55.868283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:55.868475 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:55.868897 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:55.868903 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:55.868906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:55.869278 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:38:55.869670 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:55.869958 (+ 288us) server_negotiation.cc:299] Negotiation successful
0924 11:38:55.870206 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 11:38:55.871694 25602 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:38:55.992256 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 25479
I20260924 11:38:56.013994 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 25613
Sep 24 11:38:56 dist-test-slave-cq7c krb5kdc[25239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.3@KRBTEST.COM
I20260924 11:38:56.031924 25755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:56.014874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39696 (local address 127.6.212.3:45371)
0924 11:38:56.015073 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:56.015078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:56.015098 (+ 20us) server_negotiation.cc:407] Connection header received
0924 11:38:56.015253 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:56.015257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:56.015333 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:56.015500 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:38:56.016942 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:56.017785 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:56.019334 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:56.019886 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:56.020007 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:56.025309 (+ 5302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:38:56.025326 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:38:56.025353 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:38:56.025395 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:38:56.028942 (+ 3547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:56.029724 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:56.029733 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:56.029736 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:56.029956 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:38:56.030376 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:38:56.030380 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:38:56.030382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:38:56.030767 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:38:56.031021 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:56.031409 (+ 388us) server_negotiation.cc:299] Negotiation successful
0924 11:38:56.031633 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 11:38:56.032787 25736 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:38:56.126632 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 25613
I20260924 11:38:56.143585 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 25252
I20260924 11:38:56.144903 25314 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:38:56.258641 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 25252
2026-09-24T11:38:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4477 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T11:38:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:38:56Z Disabled control of system clock
I20260924 11:38:56.340204 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:39489
--webserver_interface=127.6.212.62
--webserver_port=0
--builtin_ntp_servers=127.6.212.20:38461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:39489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 11:38:56.692924 25795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:38:56.693233 25795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:38:56.693315 25795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:38:56.700783 25795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:38:56.700906 25795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:38:56.700945 25795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:38:56.700996 25795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:38:56.701042 25795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:38:56.710608 25795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:38461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:39489
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:39489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.212.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:38:56.712076 25795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:38:56.713831 25795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:38:56.726260 25801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:38:56.726189 25803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:38:56.726693 25800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:38:56.727640 25795 server_base.cc:1034] running on GCE node
I20260924 11:38:56.728601 25795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:38:56.730180 25795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:38:56.731491 25795 hybrid_clock.cc:648] HybridClock initialized: now 1790249936731420 us; error 44 us; skew 500 ppm
I20260924 11:38:56.731901 25795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:38:56.740657 25795 webserver.cc:460] Webserver started at http://127.6.212.62:34039/ using document root <none> and password file <none>
I20260924 11:38:56.741573 25795 fs_manager.cc:359] Metadata directory not provided
I20260924 11:38:56.741727 25795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:38:56.742089 25795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:38:56.745424 25795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "788294bd0bbb48efae821be1b4b60bfa"
format_stamp: "Formatted at 2026-09-24 11:38:56 on dist-test-slave-cq7c"
server_key: "d9e8ade4c4f2b21c0c0686f34d1a46d4"
server_key_iv: "2e73c56cc42678126717129006c4fbca"
server_key_version: "encryptionkey@0"
I20260924 11:38:56.746322 25795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "788294bd0bbb48efae821be1b4b60bfa"
format_stamp: "Formatted at 2026-09-24 11:38:56 on dist-test-slave-cq7c"
server_key: "d9e8ade4c4f2b21c0c0686f34d1a46d4"
server_key_iv: "2e73c56cc42678126717129006c4fbca"
server_key_version: "encryptionkey@0"
I20260924 11:38:56.752221 25795 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 11:38:56.755908 25809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:56.757390 25795 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260924 11:38:56.757587 25795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "788294bd0bbb48efae821be1b4b60bfa"
format_stamp: "Formatted at 2026-09-24 11:38:56 on dist-test-slave-cq7c"
server_key: "d9e8ade4c4f2b21c0c0686f34d1a46d4"
server_key_iv: "2e73c56cc42678126717129006c4fbca"
server_key_version: "encryptionkey@0"
I20260924 11:38:56.757826 25795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:38:56.779292 25795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:38:56.780329 25795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:38:56.780690 25795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:38:56.801301 25795 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.62:39489
I20260924 11:38:56.801321 25860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.62:39489 every 8 connection(s)
I20260924 11:38:56.802878 25795 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:38:56.806766 25861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:38:56.808645 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 25795
I20260924 11:38:56.808820 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:38:56.809391 6992 external_mini_cluster.cc:1428] Setting key f3c287ceeed89836262cacd967306cfe
I20260924 11:38:56.819511 25861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 788294bd0bbb48efae821be1b4b60bfa: Bootstrap starting.
I20260924 11:38:56.823263 25861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 788294bd0bbb48efae821be1b4b60bfa: Neither blocks nor log segments found. Creating new log.
I20260924 11:38:56.824379 25861 log.cc:824] T 00000000000000000000000000000000 P 788294bd0bbb48efae821be1b4b60bfa: Log is configured to *not* fsync() on all Append() calls
I20260924 11:38:56.824420 25864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:56.811414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52452 (local address 127.6.212.62:39489)
0924 11:38:56.812631 (+ 1217us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:56.812643 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:56.812690 (+ 47us) server_negotiation.cc:407] Connection header received
0924 11:38:56.814591 (+ 1901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:56.814614 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:56.815135 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:56.815581 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:38:56.816116 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:56.817620 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:56.819516 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:56.820389 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:56.820533 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:56.821833 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:38:56.821843 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:38:56.821863 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:38:56.822456 (+ 593us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:38:56.822557 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:38:56.822585 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:38:56.822676 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:56.822921 (+ 245us) server_negotiation.cc:299] Negotiation successful
0924 11:38:56.823303 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":738,"thread_start_us":484,"threads_started":1}
I20260924 11:38:56.827718 25861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 788294bd0bbb48efae821be1b4b60bfa: No bootstrap required, opened a new log
I20260924 11:38:56.835116 25861 raft_consensus.cc:348] T 00000000000000000000000000000000 P 788294bd0bbb48efae821be1b4b60bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788294bd0bbb48efae821be1b4b60bfa" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 39489 } }
I20260924 11:38:56.835585 25861 raft_consensus.cc:374] T 00000000000000000000000000000000 P 788294bd0bbb48efae821be1b4b60bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:38:56.835718 25861 raft_consensus.cc:729] T 00000000000000000000000000000000 P 788294bd0bbb48efae821be1b4b60bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 788294bd0bbb48efae821be1b4b60bfa, State: Initialized, Role: FOLLOWER
I20260924 11:38:56.836355 25861 consensus_queue.cc:260] T 00000000000000000000000000000000 P 788294bd0bbb48efae821be1b4b60bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788294bd0bbb48efae821be1b4b60bfa" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 39489 } }
I20260924 11:38:56.836588 25861 raft_consensus.cc:388] T 00000000000000000000000000000000 P 788294bd0bbb48efae821be1b4b60bfa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:38:56.836714 25861 raft_consensus.cc:482] T 00000000000000000000000000000000 P 788294bd0bbb48efae821be1b4b60bfa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:38:56.836895 25861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 788294bd0bbb48efae821be1b4b60bfa [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:38:56.839058 25861 raft_consensus.cc:504] T 00000000000000000000000000000000 P 788294bd0bbb48efae821be1b4b60bfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788294bd0bbb48efae821be1b4b60bfa" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 39489 } }
I20260924 11:38:56.839742 25861 leader_election.cc:302] T 00000000000000000000000000000000 P 788294bd0bbb48efae821be1b4b60bfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 788294bd0bbb48efae821be1b4b60bfa; no voters:
I20260924 11:38:56.840387 25861 leader_election.cc:288] T 00000000000000000000000000000000 P 788294bd0bbb48efae821be1b4b60bfa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:38:56.840626 25866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 788294bd0bbb48efae821be1b4b60bfa [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:38:56.841571 25866 raft_consensus.cc:686] T 00000000000000000000000000000000 P 788294bd0bbb48efae821be1b4b60bfa [term 1 LEADER]: Becoming Leader. State: Replica: 788294bd0bbb48efae821be1b4b60bfa, State: Running, Role: LEADER
I20260924 11:38:56.842141 25866 consensus_queue.cc:237] T 00000000000000000000000000000000 P 788294bd0bbb48efae821be1b4b60bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788294bd0bbb48efae821be1b4b60bfa" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 39489 } }
I20260924 11:38:56.842700 25861 sys_catalog.cc:564] T 00000000000000000000000000000000 P 788294bd0bbb48efae821be1b4b60bfa [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:38:56.844717 25868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 788294bd0bbb48efae821be1b4b60bfa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 788294bd0bbb48efae821be1b4b60bfa. Latest consensus state: current_term: 1 leader_uuid: "788294bd0bbb48efae821be1b4b60bfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788294bd0bbb48efae821be1b4b60bfa" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 39489 } } }
I20260924 11:38:56.845161 25868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 788294bd0bbb48efae821be1b4b60bfa [sys.catalog]: This master's current role is: LEADER
I20260924 11:38:56.845795 25867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 788294bd0bbb48efae821be1b4b60bfa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "788294bd0bbb48efae821be1b4b60bfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788294bd0bbb48efae821be1b4b60bfa" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 39489 } } }
I20260924 11:38:56.846098 25867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 788294bd0bbb48efae821be1b4b60bfa [sys.catalog]: This master's current role is: LEADER
I20260924 11:38:56.847090 25872 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:38:56.853367 25872 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:38:56.863513 25872 catalog_manager.cc:1347] Generated new cluster ID: a943f14166f6468bbefdbbcf2445419a
I20260924 11:38:56.863658 25872 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:38:56.905869 25872 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:38:56.907872 25872 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:38:56.923517 25872 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 788294bd0bbb48efae821be1b4b60bfa: Generated new TSK 0
I20260924 11:38:56.924922 25872 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:38:56.943286 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.1:0
--local_ip_for_outbound_sockets=127.6.212.1
--webserver_interface=127.6.212.1
--webserver_port=0
--tserver_master_addrs=127.6.212.62:39489
--builtin_ntp_servers=127.6.212.20:38461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 11:38:57.291303 25885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:38:57.291649 25885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:38:57.291811 25885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:38:57.299144 25885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:38:57.299247 25885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:38:57.299449 25885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.1
I20260924 11:38:57.308879 25885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:38461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.212.1
--webserver_port=0
--tserver_master_addrs=127.6.212.62:39489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.1
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:38:57.310195 25885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:38:57.312085 25885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:38:57.325712 25890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:38:57.325661 25893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:38:57.325686 25891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:38:57.326686 25885 server_base.cc:1034] running on GCE node
I20260924 11:38:57.327272 25885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:38:57.328265 25885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:38:57.329494 25885 hybrid_clock.cc:648] HybridClock initialized: now 1790249937329420 us; error 63 us; skew 500 ppm
I20260924 11:38:57.329849 25885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:38:57.332216 25885 webserver.cc:460] Webserver started at http://127.6.212.1:46013/ using document root <none> and password file <none>
I20260924 11:38:57.332945 25885 fs_manager.cc:359] Metadata directory not provided
I20260924 11:38:57.333062 25885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:38:57.333425 25885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:38:57.336428 25885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-0/data/instance:
uuid: "a8de9dde77994a2daccaf6f29c2462e5"
format_stamp: "Formatted at 2026-09-24 11:38:57 on dist-test-slave-cq7c"
server_key: "9b776edf4fa4a5537026d81627b5c9f6"
server_key_iv: "43a7fbe7918a7c05423bea1193a7ef81"
server_key_version: "encryptionkey@0"
I20260924 11:38:57.337225 25885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance:
uuid: "a8de9dde77994a2daccaf6f29c2462e5"
format_stamp: "Formatted at 2026-09-24 11:38:57 on dist-test-slave-cq7c"
server_key: "9b776edf4fa4a5537026d81627b5c9f6"
server_key_iv: "43a7fbe7918a7c05423bea1193a7ef81"
server_key_version: "encryptionkey@0"
I20260924 11:38:57.342295 25885 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 11:38:57.346480 25899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:57.347990 25885 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:38:57.348156 25885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-0/wal
uuid: "a8de9dde77994a2daccaf6f29c2462e5"
format_stamp: "Formatted at 2026-09-24 11:38:57 on dist-test-slave-cq7c"
server_key: "9b776edf4fa4a5537026d81627b5c9f6"
server_key_iv: "43a7fbe7918a7c05423bea1193a7ef81"
server_key_version: "encryptionkey@0"
I20260924 11:38:57.348430 25885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:38:57.373540 25885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:38:57.374423 25885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:38:57.374720 25885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:38:57.375814 25885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:38:57.377344 25885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:38:57.377434 25885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:57.377543 25885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:38:57.377615 25885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:57.412147 25885 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.1:38953
I20260924 11:38:57.412253 26011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.1:38953 every 8 connection(s)
I20260924 11:38:57.413539 25885 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
I20260924 11:38:57.421186 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 25885
I20260924 11:38:57.421387 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance
I20260924 11:38:57.421903 6992 external_mini_cluster.cc:1428] Setting key b15d44f5658e8f795a0cf23c0d9fe3dc
I20260924 11:38:57.425747 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.2:0
--local_ip_for_outbound_sockets=127.6.212.2
--webserver_interface=127.6.212.2
--webserver_port=0
--tserver_master_addrs=127.6.212.62:39489
--builtin_ntp_servers=127.6.212.20:38461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 11:38:57.428155 26014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:57.416321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:38311 (local address 127.6.212.62:39489)
0924 11:38:57.416779 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:57.416785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:57.418095 (+ 1310us) server_negotiation.cc:407] Connection header received
0924 11:38:57.419944 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:57.419949 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:57.420017 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:57.420106 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:38:57.421223 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:57.422406 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:57.424690 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:57.425367 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:57.425522 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:57.427074 (+ 1552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:38:57.427082 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:38:57.427090 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:38:57.427115 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:38:57.427169 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:38:57.427188 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:38:57.427286 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:57.427568 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 11:38:57.427683 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":208,"threads_started":1}
I20260924 11:38:57.429052 26015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:57.416980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:39489 (local address 127.6.212.1:38311)
0924 11:38:57.417938 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:57.417979 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 11:38:57.419663 (+ 1684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:38:57.420300 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:38:57.420307 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:38:57.420396 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:38:57.421028 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:57.421046 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:57.422652 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:57.422658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:57.424518 (+ 1860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:57.424534 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:57.425596 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:57.425601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:57.425810 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:57.426630 (+ 820us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:38:57.426636 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:38:57.426695 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:38:57.426813 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:38:57.427290 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:38:57.427298 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:38:57.427304 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 11:38:57.427532 (+ 228us) client_negotiation.cc:240] Negotiation successful
0924 11:38:57.427848 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":275,"threads_started":1}
I20260924 11:38:57.430613 26012 heartbeater.cc:347] Connected to a master server at 127.6.212.62:39489
I20260924 11:38:57.430943 26012 heartbeater.cc:464] Registering TS with master...
I20260924 11:38:57.431762 26012 heartbeater.cc:511] Master 127.6.212.62:39489 requested a full tablet report, sending...
I20260924 11:38:57.433760 25826 ts_manager.cc:194] Registered new tserver with Master: a8de9dde77994a2daccaf6f29c2462e5 (127.6.212.1:38953)
I20260924 11:38:57.435173 25826 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.212.1:38311
W20260924 11:38:57.765007 26016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:38:57.765291 26016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:38:57.765434 26016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:38:57.772799 26016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:38:57.772905 26016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:38:57.773093 26016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.2
I20260924 11:38:57.782359 26016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:38461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.212.2
--webserver_port=0
--tserver_master_addrs=127.6.212.62:39489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.2
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:38:57.783731 26016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:38:57.785696 26016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:38:57.798959 26021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:38:57.798871 26024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:38:57.799013 26022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:38:57.800196 26016 server_base.cc:1034] running on GCE node
I20260924 11:38:57.800956 26016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:38:57.802150 26016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:38:57.803422 26016 hybrid_clock.cc:648] HybridClock initialized: now 1790249937803350 us; error 60 us; skew 500 ppm
I20260924 11:38:57.803839 26016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:38:57.806564 26016 webserver.cc:460] Webserver started at http://127.6.212.2:46773/ using document root <none> and password file <none>
I20260924 11:38:57.807448 26016 fs_manager.cc:359] Metadata directory not provided
I20260924 11:38:57.807569 26016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:38:57.807905 26016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:38:57.811679 26016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-1/data/instance:
uuid: "4aada0d065b34b91ae7c5d7f3959f7a0"
format_stamp: "Formatted at 2026-09-24 11:38:57 on dist-test-slave-cq7c"
server_key: "ffb563437f47757e6b12d0cb4a3d273e"
server_key_iv: "94fd0194cdfc03aa4013457597c33b20"
server_key_version: "encryptionkey@0"
I20260924 11:38:57.812618 26016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance:
uuid: "4aada0d065b34b91ae7c5d7f3959f7a0"
format_stamp: "Formatted at 2026-09-24 11:38:57 on dist-test-slave-cq7c"
server_key: "ffb563437f47757e6b12d0cb4a3d273e"
server_key_iv: "94fd0194cdfc03aa4013457597c33b20"
server_key_version: "encryptionkey@0"
I20260924 11:38:57.819119 26016 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 11:38:57.823460 26030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:57.824877 26016 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:38:57.825004 26016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-1/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-1/wal
uuid: "4aada0d065b34b91ae7c5d7f3959f7a0"
format_stamp: "Formatted at 2026-09-24 11:38:57 on dist-test-slave-cq7c"
server_key: "ffb563437f47757e6b12d0cb4a3d273e"
server_key_iv: "94fd0194cdfc03aa4013457597c33b20"
server_key_version: "encryptionkey@0"
I20260924 11:38:57.825183 26016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:38:57.846123 26016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:38:57.846941 26016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:38:57.847196 26016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:38:57.848284 26016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:38:57.850008 26016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:38:57.850096 26016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:57.850168 26016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:38:57.850222 26016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:57.885291 26016 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.2:46307
I20260924 11:38:57.885397 26142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.2:46307 every 8 connection(s)
I20260924 11:38:57.886654 26016 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
I20260924 11:38:57.893580 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 26016
I20260924 11:38:57.893833 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance
I20260924 11:38:57.894405 6992 external_mini_cluster.cc:1428] Setting key d59f4969556d5f544138fae160170d14
I20260924 11:38:57.897670 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.3:0
--local_ip_for_outbound_sockets=127.6.212.3
--webserver_interface=127.6.212.3
--webserver_port=0
--tserver_master_addrs=127.6.212.62:39489
--builtin_ntp_servers=127.6.212.20:38461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 11:38:57.900682 26014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:57.889380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:55843 (local address 127.6.212.62:39489)
0924 11:38:57.889615 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:57.889620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:57.891022 (+ 1402us) server_negotiation.cc:407] Connection header received
0924 11:38:57.892831 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:57.892837 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:57.892914 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:57.893009 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:38:57.894186 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:57.895195 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:57.897413 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:57.898097 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:57.898242 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:57.899672 (+ 1430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:38:57.899680 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:38:57.899688 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:38:57.899713 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:38:57.899757 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:38:57.899776 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:38:57.899875 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:57.900153 (+ 278us) server_negotiation.cc:299] Negotiation successful
0924 11:38:57.900268 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260924 11:38:57.901763 26145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:57.889897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:39489 (local address 127.6.212.2:55843)
0924 11:38:57.890869 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:57.890906 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 11:38:57.892553 (+ 1647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:38:57.893194 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:38:57.893201 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:38:57.893290 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:38:57.894015 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:57.894034 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:57.895439 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:57.895446 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:57.897236 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:57.897248 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:57.898305 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:57.898310 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:57.898512 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:57.899272 (+ 760us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:38:57.899274 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:38:57.899318 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:38:57.899464 (+ 146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:38:57.899919 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:38:57.899928 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:38:57.899935 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 11:38:57.900090 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 11:38:57.900425 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":287,"threads_started":1}
I20260924 11:38:57.903220 26143 heartbeater.cc:347] Connected to a master server at 127.6.212.62:39489
I20260924 11:38:57.903602 26143 heartbeater.cc:464] Registering TS with master...
I20260924 11:38:57.904547 26143 heartbeater.cc:511] Master 127.6.212.62:39489 requested a full tablet report, sending...
I20260924 11:38:57.906154 25826 ts_manager.cc:194] Registered new tserver with Master: 4aada0d065b34b91ae7c5d7f3959f7a0 (127.6.212.2:46307)
I20260924 11:38:57.907251 25826 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.212.2:55843
W20260924 11:38:58.236392 26146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:38:58.236680 26146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:38:58.236843 26146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:38:58.244529 26146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:38:58.244634 26146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:38:58.244832 26146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.3
I20260924 11:38:58.254709 26146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:38461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.212.3
--webserver_port=0
--tserver_master_addrs=127.6.212.62:39489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.3
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:38:58.256139 26146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:38:58.257946 26146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:38:58.272300 26151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:38:58.272248 26152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:38:58.272310 26154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:38:58.273448 26146 server_base.cc:1034] running on GCE node
I20260924 11:38:58.274161 26146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:38:58.275326 26146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:38:58.276636 26146 hybrid_clock.cc:648] HybridClock initialized: now 1790249938276571 us; error 58 us; skew 500 ppm
I20260924 11:38:58.276994 26146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:38:58.279598 26146 webserver.cc:460] Webserver started at http://127.6.212.3:41689/ using document root <none> and password file <none>
I20260924 11:38:58.280330 26146 fs_manager.cc:359] Metadata directory not provided
I20260924 11:38:58.280452 26146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:38:58.280804 26146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:38:58.283824 26146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-2/data/instance:
uuid: "79583e6358304074ae7e51fbf6515fd7"
format_stamp: "Formatted at 2026-09-24 11:38:58 on dist-test-slave-cq7c"
server_key: "a207e9ed5f3916f2b447d4e2ab3828cf"
server_key_iv: "5e606b31f7a58cce82b316e9495399b4"
server_key_version: "encryptionkey@0"
I20260924 11:38:58.284641 26146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance:
uuid: "79583e6358304074ae7e51fbf6515fd7"
format_stamp: "Formatted at 2026-09-24 11:38:58 on dist-test-slave-cq7c"
server_key: "a207e9ed5f3916f2b447d4e2ab3828cf"
server_key_iv: "5e606b31f7a58cce82b316e9495399b4"
server_key_version: "encryptionkey@0"
I20260924 11:38:58.289983 26146 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 11:38:58.293798 26160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:58.295352 26146 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 11:38:58.295552 26146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-2/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-2/wal
uuid: "79583e6358304074ae7e51fbf6515fd7"
format_stamp: "Formatted at 2026-09-24 11:38:58 on dist-test-slave-cq7c"
server_key: "a207e9ed5f3916f2b447d4e2ab3828cf"
server_key_iv: "5e606b31f7a58cce82b316e9495399b4"
server_key_version: "encryptionkey@0"
I20260924 11:38:58.295744 26146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:38:58.309069 26146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:38:58.309953 26146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:38:58.310195 26146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:38:58.311221 26146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:38:58.312860 26146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:38:58.312952 26146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:58.313031 26146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:38:58.313087 26146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:38:58.348425 26146 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.3:39133
I20260924 11:38:58.348555 26272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.3:39133 every 8 connection(s)
I20260924 11:38:58.349889 26146 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
I20260924 11:38:58.356530 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 26146
I20260924 11:38:58.356730 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance
I20260924 11:38:58.357329 6992 external_mini_cluster.cc:1428] Setting key 882dc3c775133cd89e6dfec8811202e5
I20260924 11:38:58.363997 26014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:58.353003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:47851 (local address 127.6.212.62:39489)
0924 11:38:58.353976 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:58.353981 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:58.354692 (+ 711us) server_negotiation.cc:407] Connection header received
0924 11:38:58.356731 (+ 2039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:58.356738 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:58.356913 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:58.357004 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:38:58.358335 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:58.359270 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:58.361043 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:58.361486 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:58.361630 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:58.363105 (+ 1475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:38:58.363111 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:38:58.363116 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:38:58.363141 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:38:58.363172 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:38:58.363184 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:38:58.363282 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:58.363568 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 11:38:58.363668 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":836}
I20260924 11:38:58.365460 26275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:58.353525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:39489 (local address 127.6.212.3:47851)
0924 11:38:58.354508 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:58.354544 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 11:38:58.356350 (+ 1806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:38:58.357289 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:38:58.357303 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:38:58.357435 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:38:58.358144 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:58.358162 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:58.359448 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:58.359453 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:58.360894 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:58.360907 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:58.361715 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:58.361723 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:58.361976 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:58.362706 (+ 730us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:38:58.362708 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:38:58.362749 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:38:58.362908 (+ 159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:38:58.363281 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:38:58.363292 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:38:58.363302 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 11:38:58.363567 (+ 265us) client_negotiation.cc:240] Negotiation successful
0924 11:38:58.364022 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":382,"threads_started":1}
I20260924 11:38:58.367236 26273 heartbeater.cc:347] Connected to a master server at 127.6.212.62:39489
I20260924 11:38:58.367594 26273 heartbeater.cc:464] Registering TS with master...
I20260924 11:38:58.368383 26273 heartbeater.cc:511] Master 127.6.212.62:39489 requested a full tablet report, sending...
I20260924 11:38:58.369779 25826 ts_manager.cc:194] Registered new tserver with Master: 79583e6358304074ae7e51fbf6515fd7 (127.6.212.3:39133)
I20260924 11:38:58.370671 25826 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.212.3:47851
I20260924 11:38:58.372953 6992 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:38:58.383508 26014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:58.377098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52468 (local address 127.6.212.62:39489)
0924 11:38:58.377386 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:58.377393 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:58.377743 (+ 350us) server_negotiation.cc:407] Connection header received
0924 11:38:58.378015 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:58.378020 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:58.378105 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:58.378216 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:38:58.378696 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:58.379454 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:58.380872 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:58.381304 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:58.381451 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:58.382607 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:38:58.382612 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:38:58.382616 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:38:58.382634 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:38:58.382665 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:38:58.382677 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:38:58.382921 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:58.383097 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 11:38:58.383153 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 11:38:58.387645 25826 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52468:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:38:58.394560 25826 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:38:58.425537 26286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:58.407303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.3:39133 (local address 127.0.0.1:60434)
0924 11:38:58.410831 (+ 3528us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:58.410845 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 11:38:58.411014 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:38:58.412766 (+ 1752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:38:58.412775 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:38:58.412810 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:38:58.413458 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:58.413475 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:58.416124 (+ 2649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:58.416132 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:58.419898 (+ 3766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:58.419920 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:58.422894 (+ 2974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:58.422903 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:58.424385 (+ 1482us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:58.424496 (+ 111us) client_negotiation.cc:769] Sending connection context
0924 11:38:58.424661 (+ 165us) client_negotiation.cc:240] Negotiation successful
0924 11:38:58.424941 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3424,"mutex_wait_us":2,"thread_start_us":3295,"threads_started":1}
I20260924 11:38:58.425653 26287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:58.407316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60434 (local address 127.6.212.3:39133)
0924 11:38:58.410597 (+ 3281us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:58.410609 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:58.410965 (+ 356us) server_negotiation.cc:407] Connection header received
0924 11:38:58.411178 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:58.411187 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:58.412151 (+ 964us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:58.412370 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:38:58.413747 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:58.415912 (+ 2165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:58.420194 (+ 4282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:58.422192 (+ 1998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:58.422359 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:58.422740 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:58.424904 (+ 2164us) server_negotiation.cc:299] Negotiation successful
0924 11:38:58.425134 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3104,"thread_start_us":2929,"threads_started":1}
I20260924 11:38:58.431154 26288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:58.407815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38572 (local address 127.6.212.1:38953)
0924 11:38:58.409593 (+ 1778us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:58.409609 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:58.409654 (+ 45us) server_negotiation.cc:407] Connection header received
0924 11:38:58.409941 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:58.409956 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:58.410340 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:58.411744 (+ 1404us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:38:58.423180 (+ 11436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:58.425014 (+ 1834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:58.428452 (+ 3438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:58.430008 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:58.430150 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:58.430448 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:58.430627 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 11:38:58.430796 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1546,"thread_start_us":1937,"threads_started":1}
I20260924 11:38:58.431172 26285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:58.407231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:38953 (local address 127.0.0.1:38572)
0924 11:38:58.408462 (+ 1231us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:58.408542 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0924 11:38:58.408805 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:38:58.410735 (+ 1930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:38:58.410745 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:38:58.410781 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:38:58.411333 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:58.411381 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:58.425213 (+ 13832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:58.425221 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:58.428215 (+ 2994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:58.428228 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:58.430204 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:58.430211 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:58.430418 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:58.430458 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 11:38:58.430552 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 11:38:58.430757 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"mutex_wait_us":36,"thread_start_us":782,"threads_started":1}
I20260924 11:38:58.430691 26208 tablet_service.cc:1467] Processing CreateTablet for tablet 629b7abe498c4804a7dcba2ed2ac5035 (DEFAULT_TABLE table=test-table [id=c29388720360487688d394c4dd533359]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:38:58.432580 26208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 629b7abe498c4804a7dcba2ed2ac5035. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:38:58.434688 25947 tablet_service.cc:1467] Processing CreateTablet for tablet 629b7abe498c4804a7dcba2ed2ac5035 (DEFAULT_TABLE table=test-table [id=c29388720360487688d394c4dd533359]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:38:58.435909 26290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:58.411333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.2:46307 (local address 127.0.0.1:59202)
0924 11:38:58.421983 (+ 10650us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:58.422007 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 11:38:58.422144 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:38:58.427445 (+ 5301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:38:58.427451 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:38:58.427483 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:38:58.428114 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:58.428133 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:58.430055 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:58.430061 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:58.433102 (+ 3041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:58.433119 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:58.435004 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:58.435011 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:58.435174 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:58.435208 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 11:38:58.435297 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 11:38:58.435587 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10499,"thread_start_us":10886,"threads_started":1}
I20260924 11:38:58.435941 26289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:58.411342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59202 (local address 127.6.212.2:46307)
0924 11:38:58.426617 (+ 15275us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:58.426626 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:58.426655 (+ 29us) server_negotiation.cc:407] Connection header received
0924 11:38:58.426748 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:58.426755 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:58.427022 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:58.427217 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:38:58.428364 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:58.429908 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:58.433355 (+ 3447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:58.434791 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:58.434928 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:58.435170 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:58.435416 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 11:38:58.435579 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15102,"thread_start_us":16143,"threads_started":1}
I20260924 11:38:58.436275 25947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 629b7abe498c4804a7dcba2ed2ac5035. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:38:58.438052 26012 heartbeater.cc:503] Master 127.6.212.62:39489 was elected leader, sending a full tablet report...
I20260924 11:38:58.440068 26078 tablet_service.cc:1467] Processing CreateTablet for tablet 629b7abe498c4804a7dcba2ed2ac5035 (DEFAULT_TABLE table=test-table [id=c29388720360487688d394c4dd533359]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:38:58.441504 26078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 629b7abe498c4804a7dcba2ed2ac5035. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:38:58.448872 26291 tablet_bootstrap.cc:492] T 629b7abe498c4804a7dcba2ed2ac5035 P 79583e6358304074ae7e51fbf6515fd7: Bootstrap starting.
I20260924 11:38:58.451061 26292 tablet_bootstrap.cc:492] T 629b7abe498c4804a7dcba2ed2ac5035 P a8de9dde77994a2daccaf6f29c2462e5: Bootstrap starting.
I20260924 11:38:58.453219 26291 tablet_bootstrap.cc:654] T 629b7abe498c4804a7dcba2ed2ac5035 P 79583e6358304074ae7e51fbf6515fd7: Neither blocks nor log segments found. Creating new log.
I20260924 11:38:58.454340 26291 log.cc:824] T 629b7abe498c4804a7dcba2ed2ac5035 P 79583e6358304074ae7e51fbf6515fd7: Log is configured to *not* fsync() on all Append() calls
I20260924 11:38:58.454762 26292 tablet_bootstrap.cc:654] T 629b7abe498c4804a7dcba2ed2ac5035 P a8de9dde77994a2daccaf6f29c2462e5: Neither blocks nor log segments found. Creating new log.
I20260924 11:38:58.456032 26292 log.cc:824] T 629b7abe498c4804a7dcba2ed2ac5035 P a8de9dde77994a2daccaf6f29c2462e5: Log is configured to *not* fsync() on all Append() calls
I20260924 11:38:58.457858 26291 tablet_bootstrap.cc:492] T 629b7abe498c4804a7dcba2ed2ac5035 P 79583e6358304074ae7e51fbf6515fd7: No bootstrap required, opened a new log
I20260924 11:38:58.457973 26294 tablet_bootstrap.cc:492] T 629b7abe498c4804a7dcba2ed2ac5035 P 4aada0d065b34b91ae7c5d7f3959f7a0: Bootstrap starting.
I20260924 11:38:58.458254 26291 ts_tablet_manager.cc:1397] T 629b7abe498c4804a7dcba2ed2ac5035 P 79583e6358304074ae7e51fbf6515fd7: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260924 11:38:58.461531 26292 tablet_bootstrap.cc:492] T 629b7abe498c4804a7dcba2ed2ac5035 P a8de9dde77994a2daccaf6f29c2462e5: No bootstrap required, opened a new log
I20260924 11:38:58.461627 26294 tablet_bootstrap.cc:654] T 629b7abe498c4804a7dcba2ed2ac5035 P 4aada0d065b34b91ae7c5d7f3959f7a0: Neither blocks nor log segments found. Creating new log.
I20260924 11:38:58.461833 26292 ts_tablet_manager.cc:1397] T 629b7abe498c4804a7dcba2ed2ac5035 P a8de9dde77994a2daccaf6f29c2462e5: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260924 11:38:58.463016 26294 log.cc:824] T 629b7abe498c4804a7dcba2ed2ac5035 P 4aada0d065b34b91ae7c5d7f3959f7a0: Log is configured to *not* fsync() on all Append() calls
I20260924 11:38:58.466230 26294 tablet_bootstrap.cc:492] T 629b7abe498c4804a7dcba2ed2ac5035 P 4aada0d065b34b91ae7c5d7f3959f7a0: No bootstrap required, opened a new log
I20260924 11:38:58.466521 26294 ts_tablet_manager.cc:1397] T 629b7abe498c4804a7dcba2ed2ac5035 P 4aada0d065b34b91ae7c5d7f3959f7a0: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.004s
I20260924 11:38:58.468855 26291 raft_consensus.cc:348] T 629b7abe498c4804a7dcba2ed2ac5035 P 79583e6358304074ae7e51fbf6515fd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79583e6358304074ae7e51fbf6515fd7" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 39133 } } peers { permanent_uuid: "4aada0d065b34b91ae7c5d7f3959f7a0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 46307 } } peers { permanent_uuid: "a8de9dde77994a2daccaf6f29c2462e5" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 38953 } }
I20260924 11:38:58.469340 26291 raft_consensus.cc:374] T 629b7abe498c4804a7dcba2ed2ac5035 P 79583e6358304074ae7e51fbf6515fd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:38:58.469439 26291 raft_consensus.cc:729] T 629b7abe498c4804a7dcba2ed2ac5035 P 79583e6358304074ae7e51fbf6515fd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79583e6358304074ae7e51fbf6515fd7, State: Initialized, Role: FOLLOWER
I20260924 11:38:58.470088 26291 consensus_queue.cc:260] T 629b7abe498c4804a7dcba2ed2ac5035 P 79583e6358304074ae7e51fbf6515fd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79583e6358304074ae7e51fbf6515fd7" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 39133 } } peers { permanent_uuid: "4aada0d065b34b91ae7c5d7f3959f7a0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 46307 } } peers { permanent_uuid: "a8de9dde77994a2daccaf6f29c2462e5" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 38953 } }
I20260924 11:38:58.470938 26292 raft_consensus.cc:348] T 629b7abe498c4804a7dcba2ed2ac5035 P a8de9dde77994a2daccaf6f29c2462e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79583e6358304074ae7e51fbf6515fd7" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 39133 } } peers { permanent_uuid: "4aada0d065b34b91ae7c5d7f3959f7a0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 46307 } } peers { permanent_uuid: "a8de9dde77994a2daccaf6f29c2462e5" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 38953 } }
I20260924 11:38:58.471261 26273 heartbeater.cc:503] Master 127.6.212.62:39489 was elected leader, sending a full tablet report...
I20260924 11:38:58.471370 26292 raft_consensus.cc:374] T 629b7abe498c4804a7dcba2ed2ac5035 P a8de9dde77994a2daccaf6f29c2462e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:38:58.471534 26292 raft_consensus.cc:729] T 629b7abe498c4804a7dcba2ed2ac5035 P a8de9dde77994a2daccaf6f29c2462e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8de9dde77994a2daccaf6f29c2462e5, State: Initialized, Role: FOLLOWER
I20260924 11:38:58.472103 26292 consensus_queue.cc:260] T 629b7abe498c4804a7dcba2ed2ac5035 P a8de9dde77994a2daccaf6f29c2462e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79583e6358304074ae7e51fbf6515fd7" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 39133 } } peers { permanent_uuid: "4aada0d065b34b91ae7c5d7f3959f7a0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 46307 } } peers { permanent_uuid: "a8de9dde77994a2daccaf6f29c2462e5" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 38953 } }
I20260924 11:38:58.473295 26291 ts_tablet_manager.cc:1428] T 629b7abe498c4804a7dcba2ed2ac5035 P 79583e6358304074ae7e51fbf6515fd7: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260924 11:38:58.476116 26294 raft_consensus.cc:348] T 629b7abe498c4804a7dcba2ed2ac5035 P 4aada0d065b34b91ae7c5d7f3959f7a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79583e6358304074ae7e51fbf6515fd7" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 39133 } } peers { permanent_uuid: "4aada0d065b34b91ae7c5d7f3959f7a0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 46307 } } peers { permanent_uuid: "a8de9dde77994a2daccaf6f29c2462e5" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 38953 } }
I20260924 11:38:58.476490 26294 raft_consensus.cc:374] T 629b7abe498c4804a7dcba2ed2ac5035 P 4aada0d065b34b91ae7c5d7f3959f7a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:38:58.476598 26294 raft_consensus.cc:729] T 629b7abe498c4804a7dcba2ed2ac5035 P 4aada0d065b34b91ae7c5d7f3959f7a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aada0d065b34b91ae7c5d7f3959f7a0, State: Initialized, Role: FOLLOWER
I20260924 11:38:58.477181 26294 consensus_queue.cc:260] T 629b7abe498c4804a7dcba2ed2ac5035 P 4aada0d065b34b91ae7c5d7f3959f7a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79583e6358304074ae7e51fbf6515fd7" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 39133 } } peers { permanent_uuid: "4aada0d065b34b91ae7c5d7f3959f7a0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 46307 } } peers { permanent_uuid: "a8de9dde77994a2daccaf6f29c2462e5" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 38953 } }
I20260924 11:38:58.478268 26143 heartbeater.cc:503] Master 127.6.212.62:39489 was elected leader, sending a full tablet report...
I20260924 11:38:58.478955 26294 ts_tablet_manager.cc:1428] T 629b7abe498c4804a7dcba2ed2ac5035 P 4aada0d065b34b91ae7c5d7f3959f7a0: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260924 11:38:58.480177 26292 ts_tablet_manager.cc:1428] T 629b7abe498c4804a7dcba2ed2ac5035 P a8de9dde77994a2daccaf6f29c2462e5: Time spent starting tablet: real 0.018s user 0.016s sys 0.003s
W20260924 11:38:58.601634 26274 tablet.cc:2369] T 629b7abe498c4804a7dcba2ed2ac5035 P 79583e6358304074ae7e51fbf6515fd7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:38:58.614897 26298 raft_consensus.cc:482] T 629b7abe498c4804a7dcba2ed2ac5035 P a8de9dde77994a2daccaf6f29c2462e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:38:58.615178 26298 raft_consensus.cc:504] T 629b7abe498c4804a7dcba2ed2ac5035 P a8de9dde77994a2daccaf6f29c2462e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79583e6358304074ae7e51fbf6515fd7" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 39133 } } peers { permanent_uuid: "4aada0d065b34b91ae7c5d7f3959f7a0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 46307 } } peers { permanent_uuid: "a8de9dde77994a2daccaf6f29c2462e5" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 38953 } }
I20260924 11:38:58.616992 26298 leader_election.cc:288] T 629b7abe498c4804a7dcba2ed2ac5035 P a8de9dde77994a2daccaf6f29c2462e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79583e6358304074ae7e51fbf6515fd7 (127.6.212.3:39133), 4aada0d065b34b91ae7c5d7f3959f7a0 (127.6.212.2:46307)
I20260924 11:38:58.624325 26289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:58.618488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:33257 (local address 127.6.212.2:46307)
0924 11:38:58.618702 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:58.618708 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:58.618932 (+ 224us) server_negotiation.cc:407] Connection header received
0924 11:38:58.619057 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:58.619062 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:58.619157 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:58.619273 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:38:58.619990 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:58.620666 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:58.622631 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:58.623439 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:58.623584 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:58.623679 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:58.623934 (+ 255us) server_negotiation.cc:299] Negotiation successful
0924 11:38:58.624060 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 11:38:58.624658 26301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:58.618221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.2:46307 (local address 127.6.212.1:33257)
0924 11:38:58.618807 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:58.618825 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:38:58.618945 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:38:58.619296 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:38:58.619300 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:38:58.619402 (+ 102us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:38:58.619824 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:58.619832 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:58.620805 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:58.620811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:58.622445 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:58.622457 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:58.623616 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:58.623622 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:58.623774 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:58.623798 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 11:38:58.623907 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 11:38:58.624041 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"mutex_wait_us":132,"spinlock_wait_cycles":768,"thread_start_us":304,"threads_started":1}
I20260924 11:38:58.624724 26287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:58.617473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:53591 (local address 127.6.212.3:39133)
0924 11:38:58.617752 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:58.617762 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:58.618304 (+ 542us) server_negotiation.cc:407] Connection header received
0924 11:38:58.618601 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:58.618608 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:58.618693 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:58.618853 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:38:58.619831 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:58.620599 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:58.622862 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:58.623759 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:58.623927 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:58.624039 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:58.624335 (+ 296us) server_negotiation.cc:299] Negotiation successful
0924 11:38:58.624464 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 11:38:58.624882 26300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:58.617397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.3:39133 (local address 127.6.212.1:53591)
0924 11:38:58.618136 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0924 11:38:58.618155 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:38:58.618390 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:38:58.619140 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:38:58.619145 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:38:58.619173 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:38:58.619651 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:58.619662 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:58.620758 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:58.620766 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:58.622644 (+ 1878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:38:58.622658 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:38:58.623970 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:38:58.623977 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:38:58.624154 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:58.624179 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 11:38:58.624307 (+ 128us) client_negotiation.cc:240] Negotiation successful
0924 11:38:58.624434 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"mutex_wait_us":97,"spinlock_wait_cycles":74880,"thread_start_us":414,"threads_started":1}
I20260924 11:38:58.626133 26098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629b7abe498c4804a7dcba2ed2ac5035" candidate_uuid: "a8de9dde77994a2daccaf6f29c2462e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aada0d065b34b91ae7c5d7f3959f7a0" is_pre_election: true
I20260924 11:38:58.626133 26228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629b7abe498c4804a7dcba2ed2ac5035" candidate_uuid: "a8de9dde77994a2daccaf6f29c2462e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79583e6358304074ae7e51fbf6515fd7" is_pre_election: true
I20260924 11:38:58.626726 26098 raft_consensus.cc:2454] T 629b7abe498c4804a7dcba2ed2ac5035 P 4aada0d065b34b91ae7c5d7f3959f7a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8de9dde77994a2daccaf6f29c2462e5 in term 0.
I20260924 11:38:58.626726 26228 raft_consensus.cc:2454] T 629b7abe498c4804a7dcba2ed2ac5035 P 79583e6358304074ae7e51fbf6515fd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8de9dde77994a2daccaf6f29c2462e5 in term 0.
I20260924 11:38:58.627493 25903 leader_election.cc:302] T 629b7abe498c4804a7dcba2ed2ac5035 P a8de9dde77994a2daccaf6f29c2462e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79583e6358304074ae7e51fbf6515fd7, a8de9dde77994a2daccaf6f29c2462e5; no voters:
I20260924 11:38:58.627859 26298 raft_consensus.cc:2781] T 629b7abe498c4804a7dcba2ed2ac5035 P a8de9dde77994a2daccaf6f29c2462e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:38:58.628008 26298 raft_consensus.cc:482] T 629b7abe498c4804a7dcba2ed2ac5035 P a8de9dde77994a2daccaf6f29c2462e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:38:58.628132 26298 raft_consensus.cc:3037] T 629b7abe498c4804a7dcba2ed2ac5035 P a8de9dde77994a2daccaf6f29c2462e5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:38:58.630544 26298 raft_consensus.cc:504] T 629b7abe498c4804a7dcba2ed2ac5035 P a8de9dde77994a2daccaf6f29c2462e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79583e6358304074ae7e51fbf6515fd7" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 39133 } } peers { permanent_uuid: "4aada0d065b34b91ae7c5d7f3959f7a0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 46307 } } peers { permanent_uuid: "a8de9dde77994a2daccaf6f29c2462e5" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 38953 } }
I20260924 11:38:58.631173 26298 leader_election.cc:288] T 629b7abe498c4804a7dcba2ed2ac5035 P a8de9dde77994a2daccaf6f29c2462e5 [CANDIDATE]: Term 1 election: Requested vote from peers 79583e6358304074ae7e51fbf6515fd7 (127.6.212.3:39133), 4aada0d065b34b91ae7c5d7f3959f7a0 (127.6.212.2:46307)
I20260924 11:38:58.631719 26228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629b7abe498c4804a7dcba2ed2ac5035" candidate_uuid: "a8de9dde77994a2daccaf6f29c2462e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79583e6358304074ae7e51fbf6515fd7"
I20260924 11:38:58.631805 26098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629b7abe498c4804a7dcba2ed2ac5035" candidate_uuid: "a8de9dde77994a2daccaf6f29c2462e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4aada0d065b34b91ae7c5d7f3959f7a0"
I20260924 11:38:58.631897 26228 raft_consensus.cc:3037] T 629b7abe498c4804a7dcba2ed2ac5035 P 79583e6358304074ae7e51fbf6515fd7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:38:58.632098 26098 raft_consensus.cc:3037] T 629b7abe498c4804a7dcba2ed2ac5035 P 4aada0d065b34b91ae7c5d7f3959f7a0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:38:58.634140 26228 raft_consensus.cc:2454] T 629b7abe498c4804a7dcba2ed2ac5035 P 79583e6358304074ae7e51fbf6515fd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8de9dde77994a2daccaf6f29c2462e5 in term 1.
I20260924 11:38:58.634426 26098 raft_consensus.cc:2454] T 629b7abe498c4804a7dcba2ed2ac5035 P 4aada0d065b34b91ae7c5d7f3959f7a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8de9dde77994a2daccaf6f29c2462e5 in term 1.
I20260924 11:38:58.634591 25903 leader_election.cc:302] T 629b7abe498c4804a7dcba2ed2ac5035 P a8de9dde77994a2daccaf6f29c2462e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79583e6358304074ae7e51fbf6515fd7, a8de9dde77994a2daccaf6f29c2462e5; no voters:
I20260924 11:38:58.634910 26298 raft_consensus.cc:2781] T 629b7abe498c4804a7dcba2ed2ac5035 P a8de9dde77994a2daccaf6f29c2462e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:38:58.635448 26298 raft_consensus.cc:686] T 629b7abe498c4804a7dcba2ed2ac5035 P a8de9dde77994a2daccaf6f29c2462e5 [term 1 LEADER]: Becoming Leader. State: Replica: a8de9dde77994a2daccaf6f29c2462e5, State: Running, Role: LEADER
I20260924 11:38:58.635957 26298 consensus_queue.cc:237] T 629b7abe498c4804a7dcba2ed2ac5035 P a8de9dde77994a2daccaf6f29c2462e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79583e6358304074ae7e51fbf6515fd7" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 39133 } } peers { permanent_uuid: "4aada0d065b34b91ae7c5d7f3959f7a0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 46307 } } peers { permanent_uuid: "a8de9dde77994a2daccaf6f29c2462e5" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 38953 } }
W20260924 11:38:58.638732 26144 tablet.cc:2369] T 629b7abe498c4804a7dcba2ed2ac5035 P 4aada0d065b34b91ae7c5d7f3959f7a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:38:58.640934 25825 catalog_manager.cc:5495] T 629b7abe498c4804a7dcba2ed2ac5035 P a8de9dde77994a2daccaf6f29c2462e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a8de9dde77994a2daccaf6f29c2462e5 (127.6.212.1). New cstate: current_term: 1 leader_uuid: "a8de9dde77994a2daccaf6f29c2462e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79583e6358304074ae7e51fbf6515fd7" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 39133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4aada0d065b34b91ae7c5d7f3959f7a0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 46307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8de9dde77994a2daccaf6f29c2462e5" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 38953 } health_report { overall_health: HEALTHY } } }
W20260924 11:38:58.666453 26013 tablet.cc:2369] T 629b7abe498c4804a7dcba2ed2ac5035 P a8de9dde77994a2daccaf6f29c2462e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:38:58.668816 26288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:58.662774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38586 (local address 127.6.212.1:38953)
0924 11:38:58.663032 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:58.663040 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:58.663060 (+ 20us) server_negotiation.cc:407] Connection header received
0924 11:38:58.663306 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:58.663311 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:58.663476 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:58.663636 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:38:58.664017 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:58.664735 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:58.666403 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:58.666955 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:58.667088 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:58.667735 (+ 647us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:38:58.668224 (+ 489us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:38:58.668414 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:58.668537 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 11:38:58.668579 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260924 11:38:58.677309 26098 raft_consensus.cc:1261] T 629b7abe498c4804a7dcba2ed2ac5035 P 4aada0d065b34b91ae7c5d7f3959f7a0 [term 1 FOLLOWER]: Refusing update from remote peer a8de9dde77994a2daccaf6f29c2462e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:38:58.677295 26228 raft_consensus.cc:1261] T 629b7abe498c4804a7dcba2ed2ac5035 P 79583e6358304074ae7e51fbf6515fd7 [term 1 FOLLOWER]: Refusing update from remote peer a8de9dde77994a2daccaf6f29c2462e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:38:58.678308 26298 consensus_queue.cc:1035] T 629b7abe498c4804a7dcba2ed2ac5035 P a8de9dde77994a2daccaf6f29c2462e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79583e6358304074ae7e51fbf6515fd7" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 39133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:38:58.678654 26302 consensus_queue.cc:1035] T 629b7abe498c4804a7dcba2ed2ac5035 P a8de9dde77994a2daccaf6f29c2462e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4aada0d065b34b91ae7c5d7f3959f7a0" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:38:58.711644 25826 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:38:58.711988 25826 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:38:58.716646 25826 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 788294bd0bbb48efae821be1b4b60bfa: Sending DeleteTablet for 3 replicas of tablet 629b7abe498c4804a7dcba2ed2ac5035
I20260924 11:38:58.718277 26078 tablet_service.cc:1514] Processing DeleteTablet for tablet 629b7abe498c4804a7dcba2ed2ac5035 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:38:58 UTC) from {username='slave'} at 127.0.0.1:59202
I20260924 11:38:58.718278 26208 tablet_service.cc:1514] Processing DeleteTablet for tablet 629b7abe498c4804a7dcba2ed2ac5035 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:38:58 UTC) from {username='slave'} at 127.0.0.1:60434
I20260924 11:38:58.718608 25947 tablet_service.cc:1514] Processing DeleteTablet for tablet 629b7abe498c4804a7dcba2ed2ac5035 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:38:58 UTC) from {username='slave'} at 127.0.0.1:38572
I20260924 11:38:58.721613 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 25885
I20260924 11:38:58.728947 26317 tablet_replica.cc:330] T 629b7abe498c4804a7dcba2ed2ac5035 P a8de9dde77994a2daccaf6f29c2462e5: stopping tablet replica
I20260924 11:38:58.729799 26317 raft_consensus.cc:2229] T 629b7abe498c4804a7dcba2ed2ac5035 P a8de9dde77994a2daccaf6f29c2462e5 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:38:58.731000 26317 raft_consensus.cc:2258] T 629b7abe498c4804a7dcba2ed2ac5035 P a8de9dde77994a2daccaf6f29c2462e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:38:58.732146 26316 tablet_replica.cc:330] T 629b7abe498c4804a7dcba2ed2ac5035 P 4aada0d065b34b91ae7c5d7f3959f7a0: stopping tablet replica
I20260924 11:38:58.732808 26316 raft_consensus.cc:2229] T 629b7abe498c4804a7dcba2ed2ac5035 P 4aada0d065b34b91ae7c5d7f3959f7a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:38:58.733546 26316 raft_consensus.cc:2258] T 629b7abe498c4804a7dcba2ed2ac5035 P 4aada0d065b34b91ae7c5d7f3959f7a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:38:58.734097 26317 ts_tablet_manager.cc:1905] T 629b7abe498c4804a7dcba2ed2ac5035 P a8de9dde77994a2daccaf6f29c2462e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:38:58.735059 26315 tablet_replica.cc:330] T 629b7abe498c4804a7dcba2ed2ac5035 P 79583e6358304074ae7e51fbf6515fd7: stopping tablet replica
I20260924 11:38:58.735633 26315 raft_consensus.cc:2229] T 629b7abe498c4804a7dcba2ed2ac5035 P 79583e6358304074ae7e51fbf6515fd7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:38:58.736083 26316 ts_tablet_manager.cc:1905] T 629b7abe498c4804a7dcba2ed2ac5035 P 4aada0d065b34b91ae7c5d7f3959f7a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:38:58.736414 26315 raft_consensus.cc:2258] T 629b7abe498c4804a7dcba2ed2ac5035 P 79583e6358304074ae7e51fbf6515fd7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:38:58.738745 26315 ts_tablet_manager.cc:1905] T 629b7abe498c4804a7dcba2ed2ac5035 P 79583e6358304074ae7e51fbf6515fd7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:38:58.743153 26316 ts_tablet_manager.cc:1918] T 629b7abe498c4804a7dcba2ed2ac5035 P 4aada0d065b34b91ae7c5d7f3959f7a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:38:58.743391 26316 log.cc:1196] T 629b7abe498c4804a7dcba2ed2ac5035 P 4aada0d065b34b91ae7c5d7f3959f7a0: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-1/wal/wals/629b7abe498c4804a7dcba2ed2ac5035
I20260924 11:38:58.743489 26317 ts_tablet_manager.cc:1918] T 629b7abe498c4804a7dcba2ed2ac5035 P a8de9dde77994a2daccaf6f29c2462e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:38:58.743677 26317 log.cc:1196] T 629b7abe498c4804a7dcba2ed2ac5035 P a8de9dde77994a2daccaf6f29c2462e5: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-0/wal/wals/629b7abe498c4804a7dcba2ed2ac5035
I20260924 11:38:58.743983 26316 ts_tablet_manager.cc:1939] T 629b7abe498c4804a7dcba2ed2ac5035 P 4aada0d065b34b91ae7c5d7f3959f7a0: Deleting consensus metadata
I20260924 11:38:58.744263 26317 ts_tablet_manager.cc:1939] T 629b7abe498c4804a7dcba2ed2ac5035 P a8de9dde77994a2daccaf6f29c2462e5: Deleting consensus metadata
I20260924 11:38:58.745997 25813 catalog_manager.cc:4841] TS 4aada0d065b34b91ae7c5d7f3959f7a0 (127.6.212.2:46307): tablet 629b7abe498c4804a7dcba2ed2ac5035 (table test-table [id=c29388720360487688d394c4dd533359]) successfully deleted
I20260924 11:38:58.746340 25810 catalog_manager.cc:4841] TS a8de9dde77994a2daccaf6f29c2462e5 (127.6.212.1:38953): tablet 629b7abe498c4804a7dcba2ed2ac5035 (table test-table [id=c29388720360487688d394c4dd533359]) successfully deleted
I20260924 11:38:58.747445 26315 ts_tablet_manager.cc:1918] T 629b7abe498c4804a7dcba2ed2ac5035 P 79583e6358304074ae7e51fbf6515fd7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:38:58.747872 26315 log.cc:1196] T 629b7abe498c4804a7dcba2ed2ac5035 P 79583e6358304074ae7e51fbf6515fd7: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790249617057452-6992-0/minicluster-data/ts-2/wal/wals/629b7abe498c4804a7dcba2ed2ac5035
I20260924 11:38:58.748544 26315 ts_tablet_manager.cc:1939] T 629b7abe498c4804a7dcba2ed2ac5035 P 79583e6358304074ae7e51fbf6515fd7: Deleting consensus metadata
I20260924 11:38:58.750890 25813 catalog_manager.cc:4841] TS 79583e6358304074ae7e51fbf6515fd7 (127.6.212.3:39133): tablet 629b7abe498c4804a7dcba2ed2ac5035 (table test-table [id=c29388720360487688d394c4dd533359]) successfully deleted
I20260924 11:38:58.756353 26288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:58.727084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38590 (local address 127.6.212.1:38953)
0924 11:38:58.727347 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:58.727372 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:58.741088 (+ 13716us) server_negotiation.cc:407] Connection header received
0924 11:38:58.741663 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:58.741669 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:58.741833 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:58.741932 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:38:58.742698 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:58.743996 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:58.751634 (+ 7638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:58.752582 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:58.752778 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:58.754291 (+ 1513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:38:58.754305 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:38:58.754332 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:38:58.755287 (+ 955us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:38:58.755392 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:38:58.755423 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:38:58.755546 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:58.755729 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 11:38:58.755872 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 11:38:58.757400 26007 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:38:58.871564 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 25885
I20260924 11:38:58.891605 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 26016
I20260924 11:38:58.898789 26289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:58.892374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59210 (local address 127.6.212.2:46307)
0924 11:38:58.892534 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:58.892541 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:58.892611 (+ 70us) server_negotiation.cc:407] Connection header received
0924 11:38:58.892805 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:58.892811 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:58.892911 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:58.893025 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:38:58.893417 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:58.894242 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:58.895640 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:58.896072 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:58.896229 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:58.897355 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:38:58.897374 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:38:58.897389 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:38:58.897997 (+ 608us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:38:58.898046 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:38:58.898065 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:38:58.898204 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:58.898332 (+ 128us) server_negotiation.cc:299] Negotiation successful
0924 11:38:58.898439 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 11:38:58.899672 26138 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:38:58.989732 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 26016
I20260924 11:38:59.007932 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 26146
I20260924 11:38:59.014905 26287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:38:59.008898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60438 (local address 127.6.212.3:39133)
0924 11:38:59.009067 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 11:38:59.009071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:38:59.009090 (+ 19us) server_negotiation.cc:407] Connection header received
0924 11:38:59.009214 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:38:59.009218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:38:59.009312 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:38:59.009413 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:38:59.009842 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:59.010590 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:59.012076 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:38:59.012475 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:38:59.012585 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:38:59.013615 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:38:59.013624 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:38:59.013640 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:38:59.014258 (+ 618us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:38:59.014309 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:38:59.014330 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:38:59.014420 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 11:38:59.014550 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 11:38:59.014639 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 11:38:59.015635 26268 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:38:59.106755 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 26146
I20260924 11:38:59.123636 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 25795
I20260924 11:38:59.124781 25856 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:38:59.235275 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 25795
2026-09-24T11:38:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2975 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:38:59 dist-test-slave-cq7c krb5kdc[26325](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:38:59 dist-test-slave-cq7c krb5kdc[26325](info): set up 2 sockets
Sep 24 11:38:59 dist-test-slave-cq7c krb5kdc[26325](info): commencing operation
krb5kdc: starting...
W20260924 11:39:00.376588 6992 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.002s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Sep 24 11:39:00 dist-test-slave-cq7c krb5kdc[26325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:39:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:39:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.212.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
I20260924 11:39:00.658857 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:40843
--webserver_interface=127.6.212.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:36407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:40843
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:39:01.025734 26338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:39:01.026153 26338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:39:01.026335 26338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:39:01.035084 26338 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:39:01.035259 26338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:39:01.035408 26338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:39:01.035532 26338 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:39:01.035622 26338 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:39:01.047272 26338 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:36407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:40843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:40843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.212.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:39:01.048892 26338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:39:01.050387 26338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:39:01.064177 26343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:01.064268 26344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:01.064307 26346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:39:01.064934 26338 server_base.cc:1034] running on GCE node
I20260924 11:39:01.065884 26338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:39:01.067579 26338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:39:01.068853 26338 hybrid_clock.cc:648] HybridClock initialized: now 1790249941068807 us; error 40 us; skew 500 ppm
I20260924 11:39:01.069305 26338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:39:01 dist-test-slave-cq7c krb5kdc[26325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249941, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:39:01.072721 26338 init.cc:377] Logged in from keytab as kudu/127.6.212.62@KRBTEST.COM (short username kudu)
I20260924 11:39:01.074488 26338 webserver.cc:460] Webserver started at http://127.6.212.62:34431/ using document root <none> and password file <none>
I20260924 11:39:01.075278 26338 fs_manager.cc:359] Metadata directory not provided
I20260924 11:39:01.075482 26338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:39:01.075834 26338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:39:01.078991 26338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "e58765b92d6e4bcf9d3673146b0e4345"
format_stamp: "Formatted at 2026-09-24 11:39:01 on dist-test-slave-cq7c"
server_key: "6929691fe907973773c5d5603f57020e"
server_key_iv: "3d75a8424ae168965ebac8cf0a9086d3"
server_key_version: "encryptionkey@0"
I20260924 11:39:01.079909 26338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "e58765b92d6e4bcf9d3673146b0e4345"
format_stamp: "Formatted at 2026-09-24 11:39:01 on dist-test-slave-cq7c"
server_key: "6929691fe907973773c5d5603f57020e"
server_key_iv: "3d75a8424ae168965ebac8cf0a9086d3"
server_key_version: "encryptionkey@0"
I20260924 11:39:01.085477 26338 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 11:39:01.089274 26353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:39:01.090847 26338 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 11:39:01.091064 26338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "e58765b92d6e4bcf9d3673146b0e4345"
format_stamp: "Formatted at 2026-09-24 11:39:01 on dist-test-slave-cq7c"
server_key: "6929691fe907973773c5d5603f57020e"
server_key_iv: "3d75a8424ae168965ebac8cf0a9086d3"
server_key_version: "encryptionkey@0"
I20260924 11:39:01.091310 26338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:39:01.107239 26338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:39:01.112216 26338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:39:01.112566 26338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:39:01.133410 26338 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.62:40843
I20260924 11:39:01.133442 26404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.62:40843 every 8 connection(s)
I20260924 11:39:01.134955 26338 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:39:01.137102 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 26338
I20260924 11:39:01.137305 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:39:01.137748 6992 external_mini_cluster.cc:1428] Setting key 43034335c32dbd1d59efff4a157d2824
I20260924 11:39:01.139900 26405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:39:01.154634 26405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e58765b92d6e4bcf9d3673146b0e4345: Bootstrap starting.
Sep 24 11:39:01 dist-test-slave-cq7c krb5kdc[26325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:39:01.159060 26405 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e58765b92d6e4bcf9d3673146b0e4345: Neither blocks nor log segments found. Creating new log.
I20260924 11:39:01.160588 26405 log.cc:824] T 00000000000000000000000000000000 P e58765b92d6e4bcf9d3673146b0e4345: Log is configured to *not* fsync() on all Append() calls
I20260924 11:39:01.164896 26405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e58765b92d6e4bcf9d3673146b0e4345: No bootstrap required, opened a new log
I20260924 11:39:01.166296 26408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:01.140039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44390 (local address 127.6.212.62:40843)
0924 11:39:01.140941 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:01.140960 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:01.141013 (+ 53us) server_negotiation.cc:407] Connection header received
0924 11:39:01.143283 (+ 2270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:01.143315 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:01.143968 (+ 653us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:01.144474 (+ 506us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:01.146086 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:01.148227 (+ 2141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:01.149714 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:01.150925 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:01.151158 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:01.156083 (+ 4925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:01.156096 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:01.156120 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:39:01.156179 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:01.160412 (+ 4233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:01.161582 (+ 1170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:01.161598 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:01.161606 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:01.161819 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:01.162353 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:01.162362 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:01.162367 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:01.163228 (+ 861us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:01.163601 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:01.164114 (+ 513us) server_negotiation.cc:299] Negotiation successful
0924 11:39:01.164656 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":353,"threads_started":1}
I20260924 11:39:01.174978 26405 raft_consensus.cc:348] T 00000000000000000000000000000000 P e58765b92d6e4bcf9d3673146b0e4345 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e58765b92d6e4bcf9d3673146b0e4345" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 40843 } }
I20260924 11:39:01.175516 26405 raft_consensus.cc:374] T 00000000000000000000000000000000 P e58765b92d6e4bcf9d3673146b0e4345 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:39:01.175618 26405 raft_consensus.cc:729] T 00000000000000000000000000000000 P e58765b92d6e4bcf9d3673146b0e4345 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e58765b92d6e4bcf9d3673146b0e4345, State: Initialized, Role: FOLLOWER
I20260924 11:39:01.176380 26405 consensus_queue.cc:260] T 00000000000000000000000000000000 P e58765b92d6e4bcf9d3673146b0e4345 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e58765b92d6e4bcf9d3673146b0e4345" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 40843 } }
I20260924 11:39:01.176682 26405 raft_consensus.cc:388] T 00000000000000000000000000000000 P e58765b92d6e4bcf9d3673146b0e4345 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:39:01.176874 26405 raft_consensus.cc:482] T 00000000000000000000000000000000 P e58765b92d6e4bcf9d3673146b0e4345 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:39:01.177091 26405 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e58765b92d6e4bcf9d3673146b0e4345 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:39:01.179664 26405 raft_consensus.cc:504] T 00000000000000000000000000000000 P e58765b92d6e4bcf9d3673146b0e4345 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e58765b92d6e4bcf9d3673146b0e4345" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 40843 } }
I20260924 11:39:01.180349 26405 leader_election.cc:302] T 00000000000000000000000000000000 P e58765b92d6e4bcf9d3673146b0e4345 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e58765b92d6e4bcf9d3673146b0e4345; no voters:
I20260924 11:39:01.181054 26405 leader_election.cc:288] T 00000000000000000000000000000000 P e58765b92d6e4bcf9d3673146b0e4345 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:39:01.181210 26410 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e58765b92d6e4bcf9d3673146b0e4345 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:39:01.182004 26410 raft_consensus.cc:686] T 00000000000000000000000000000000 P e58765b92d6e4bcf9d3673146b0e4345 [term 1 LEADER]: Becoming Leader. State: Replica: e58765b92d6e4bcf9d3673146b0e4345, State: Running, Role: LEADER
I20260924 11:39:01.182569 26410 consensus_queue.cc:237] T 00000000000000000000000000000000 P e58765b92d6e4bcf9d3673146b0e4345 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e58765b92d6e4bcf9d3673146b0e4345" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 40843 } }
I20260924 11:39:01.183578 26405 sys_catalog.cc:564] T 00000000000000000000000000000000 P e58765b92d6e4bcf9d3673146b0e4345 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:39:01.185881 26411 sys_catalog.cc:455] T 00000000000000000000000000000000 P e58765b92d6e4bcf9d3673146b0e4345 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e58765b92d6e4bcf9d3673146b0e4345" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e58765b92d6e4bcf9d3673146b0e4345" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 40843 } } }
I20260924 11:39:01.185959 26412 sys_catalog.cc:455] T 00000000000000000000000000000000 P e58765b92d6e4bcf9d3673146b0e4345 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e58765b92d6e4bcf9d3673146b0e4345. Latest consensus state: current_term: 1 leader_uuid: "e58765b92d6e4bcf9d3673146b0e4345" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e58765b92d6e4bcf9d3673146b0e4345" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 40843 } } }
I20260924 11:39:01.186410 26412 sys_catalog.cc:458] T 00000000000000000000000000000000 P e58765b92d6e4bcf9d3673146b0e4345 [sys.catalog]: This master's current role is: LEADER
I20260924 11:39:01.187333 26416 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:39:01.186280 26411 sys_catalog.cc:458] T 00000000000000000000000000000000 P e58765b92d6e4bcf9d3673146b0e4345 [sys.catalog]: This master's current role is: LEADER
I20260924 11:39:01.192822 26416 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:39:01.204538 26416 catalog_manager.cc:1347] Generated new cluster ID: 95c6700345c04ee68462a8e3f9c3c581
I20260924 11:39:01.204672 26416 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:39:01.217280 26416 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:39:01.219286 26416 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:39:01.229008 26416 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e58765b92d6e4bcf9d3673146b0e4345: Generated new TSK 0
I20260924 11:39:01.230178 26416 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.212.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab.
Entry for principal kudu/127.6.212.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab.
I20260924 11:39:01.304276 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.1:0
--local_ip_for_outbound_sockets=127.6.212.1
--webserver_interface=127.6.212.1
--webserver_port=0
--tserver_master_addrs=127.6.212.62:40843
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab
--principal=kudu/127.6.212.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:36407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:39:01.665798 26431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:39:01.666152 26431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:39:01.666340 26431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:39:01.673856 26431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:39:01.674048 26431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:39:01.674229 26431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.1
I20260924 11:39:01.684880 26431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:36407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.1.keytab
--principal=kudu/127.6.212.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.212.1
--webserver_port=0
--tserver_master_addrs=127.6.212.62:40843
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.1
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:39:01.686295 26431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:39:01.687808 26431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:39:01.701954 26437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:01.701987 26439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:01.702034 26436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:39:01.705165 26431 server_base.cc:1034] running on GCE node
I20260924 11:39:01.706101 26431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:39:01.707511 26431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:39:01.708732 26431 hybrid_clock.cc:648] HybridClock initialized: now 1790249941708642 us; error 68 us; skew 500 ppm
I20260924 11:39:01.709144 26431 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:39:01 dist-test-slave-cq7c krb5kdc[26325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249941, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:39:01.713837 26431 init.cc:377] Logged in from keytab as kudu/127.6.212.1@KRBTEST.COM (short username kudu)
I20260924 11:39:01.715683 26431 webserver.cc:460] Webserver started at http://127.6.212.1:36619/ using document root <none> and password file <none>
I20260924 11:39:01.716475 26431 fs_manager.cc:359] Metadata directory not provided
I20260924 11:39:01.716611 26431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:39:01.716954 26431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:39:01.720049 26431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-0/data/instance:
uuid: "dfea1e60e88d440c828958db825fa574"
format_stamp: "Formatted at 2026-09-24 11:39:01 on dist-test-slave-cq7c"
server_key: "44b2c3cf625b357b10501398b02a47cd"
server_key_iv: "810e8da372dec17253f1bbaff731c8d4"
server_key_version: "encryptionkey@0"
I20260924 11:39:01.720866 26431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance:
uuid: "dfea1e60e88d440c828958db825fa574"
format_stamp: "Formatted at 2026-09-24 11:39:01 on dist-test-slave-cq7c"
server_key: "44b2c3cf625b357b10501398b02a47cd"
server_key_iv: "810e8da372dec17253f1bbaff731c8d4"
server_key_version: "encryptionkey@0"
I20260924 11:39:01.726795 26431 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260924 11:39:01.730702 26446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:39:01.732558 26431 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 11:39:01.732740 26431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-0/wal
uuid: "dfea1e60e88d440c828958db825fa574"
format_stamp: "Formatted at 2026-09-24 11:39:01 on dist-test-slave-cq7c"
server_key: "44b2c3cf625b357b10501398b02a47cd"
server_key_iv: "810e8da372dec17253f1bbaff731c8d4"
server_key_version: "encryptionkey@0"
I20260924 11:39:01.733001 26431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:39:01.773890 26431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:39:01.779038 26431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:39:01.779410 26431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:39:01.780580 26431 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:39:01.782127 26431 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:39:01.782212 26431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:39:01.782375 26431 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:39:01.782444 26431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:39:01.819527 26431 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.1:36489
I20260924 11:39:01.819658 26558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.1:36489 every 8 connection(s)
I20260924 11:39:01.820978 26431 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
I20260924 11:39:01.825623 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 26431
I20260924 11:39:01.825819 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance
I20260924 11:39:01.826366 6992 external_mini_cluster.cc:1428] Setting key 6e98e9e548711f513a7a39b29a006de7
Sep 24 11:39:01 dist-test-slave-cq7c krb5kdc[26325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249941, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.1@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:39:01.845579 26561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:01.823852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:36575 (local address 127.6.212.62:40843)
0924 11:39:01.824359 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:01.824364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:01.825388 (+ 1024us) server_negotiation.cc:407] Connection header received
0924 11:39:01.827964 (+ 2576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:01.827969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:01.828045 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:01.828139 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:01.830398 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:01.831120 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:01.833892 (+ 2772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:01.834380 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:01.834523 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:01.839342 (+ 4819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:01.839348 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:01.839352 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:39:01.839445 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:01.842131 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:01.842994 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:01.842999 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:01.843001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:01.843125 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:01.843673 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:01.843682 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:01.843685 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:01.843974 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:01.844142 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:01.844951 (+ 809us) server_negotiation.cc:299] Negotiation successful
0924 11:39:01.845199 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":262,"threads_started":1}
I20260924 11:39:01.846674 26562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:01.824370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:40843 (local address 127.6.212.1:36575)
0924 11:39:01.825175 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:01.825229 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 11:39:01.827625 (+ 2396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:39:01.828439 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:39:01.828448 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:39:01.829249 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:39:01.830159 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:01.830190 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:01.831321 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:01.831330 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:01.833678 (+ 2348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:01.833697 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:01.834657 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:01.834666 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:01.835031 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:01.835894 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:39:01.835938 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:39:01.839048 (+ 3110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:39:01.842318 (+ 3270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:39:01.842327 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:39:01.842347 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:39:01.842838 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:39:01.843246 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:39:01.843251 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:39:01.843255 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:39:01.843545 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:39:01.844112 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:39:01.844120 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:39:01.844525 (+ 405us) client_negotiation.cc:769] Sending connection context
0924 11:39:01.844828 (+ 303us) client_negotiation.cc:240] Negotiation successful
0924 11:39:01.845444 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":289,"threads_started":1}
I20260924 11:39:01.848563 26559 heartbeater.cc:347] Connected to a master server at 127.6.212.62:40843
I20260924 11:39:01.848912 26559 heartbeater.cc:464] Registering TS with master...
I20260924 11:39:01.849700 26559 heartbeater.cc:511] Master 127.6.212.62:40843 requested a full tablet report, sending...
I20260924 11:39:01.852505 26370 ts_manager.cc:194] Registered new tserver with Master: dfea1e60e88d440c828958db825fa574 (127.6.212.1:36489)
I20260924 11:39:01.854938 26370 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.1@KRBTEST.COM'} at 127.6.212.1:36575
WARNING: no policy specified for kudu/127.6.212.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab.
Entry for principal kudu/127.6.212.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab.
I20260924 11:39:01.892848 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.2:0
--local_ip_for_outbound_sockets=127.6.212.2
--webserver_interface=127.6.212.2
--webserver_port=0
--tserver_master_addrs=127.6.212.62:40843
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab
--principal=kudu/127.6.212.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:36407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:39:02.249830 26565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:39:02.250145 26565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:39:02.250367 26565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:39:02.258036 26565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:39:02.258172 26565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:39:02.258384 26565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.2
I20260924 11:39:02.268950 26565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:36407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.2.keytab
--principal=kudu/127.6.212.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.212.2
--webserver_port=0
--tserver_master_addrs=127.6.212.62:40843
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.2
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:39:02.270334 26565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:39:02.271832 26565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:39:02.285658 26573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:02.285780 26570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:02.285835 26571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:39:02.286861 26565 server_base.cc:1034] running on GCE node
I20260924 11:39:02.287575 26565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:39:02.288554 26565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:39:02.289799 26565 hybrid_clock.cc:648] HybridClock initialized: now 1790249942289707 us; error 58 us; skew 500 ppm
I20260924 11:39:02.290139 26565 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:39:02 dist-test-slave-cq7c krb5kdc[26325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249942, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:39:02.293473 26565 init.cc:377] Logged in from keytab as kudu/127.6.212.2@KRBTEST.COM (short username kudu)
I20260924 11:39:02.295112 26565 webserver.cc:460] Webserver started at http://127.6.212.2:38939/ using document root <none> and password file <none>
I20260924 11:39:02.295989 26565 fs_manager.cc:359] Metadata directory not provided
I20260924 11:39:02.296135 26565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:39:02.296475 26565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:39:02.299744 26565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-1/data/instance:
uuid: "7cbccf015b054d999764ce9812020d63"
format_stamp: "Formatted at 2026-09-24 11:39:02 on dist-test-slave-cq7c"
server_key: "2a882501f449658fd039b283b1aa9ba8"
server_key_iv: "d0a29974c8eb5804921a2ff3eba7ef14"
server_key_version: "encryptionkey@0"
I20260924 11:39:02.300606 26565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance:
uuid: "7cbccf015b054d999764ce9812020d63"
format_stamp: "Formatted at 2026-09-24 11:39:02 on dist-test-slave-cq7c"
server_key: "2a882501f449658fd039b283b1aa9ba8"
server_key_iv: "d0a29974c8eb5804921a2ff3eba7ef14"
server_key_version: "encryptionkey@0"
I20260924 11:39:02.305919 26565 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 11:39:02.309513 26580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:39:02.310925 26565 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260924 11:39:02.311093 26565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-1/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-1/wal
uuid: "7cbccf015b054d999764ce9812020d63"
format_stamp: "Formatted at 2026-09-24 11:39:02 on dist-test-slave-cq7c"
server_key: "2a882501f449658fd039b283b1aa9ba8"
server_key_iv: "d0a29974c8eb5804921a2ff3eba7ef14"
server_key_version: "encryptionkey@0"
I20260924 11:39:02.311326 26565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:39:02.329964 26565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:39:02.334815 26565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:39:02.335144 26565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:39:02.336234 26565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:39:02.337889 26565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:39:02.338002 26565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:39:02.338157 26565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:39:02.338229 26565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:39:02.372736 26565 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.2:45231
I20260924 11:39:02.372785 26692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.2:45231 every 8 connection(s)
I20260924 11:39:02.374068 26565 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
I20260924 11:39:02.382306 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 26565
I20260924 11:39:02.382506 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance
I20260924 11:39:02.383013 6992 external_mini_cluster.cc:1428] Setting key 00a20f2bde634fa5fa1398a99b80b182
Sep 24 11:39:02 dist-test-slave-cq7c krb5kdc[26325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249942, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.2@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:39:02.397755 26695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:02.376824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:39919 (local address 127.6.212.62:40843)
0924 11:39:02.377322 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:02.377329 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:02.378438 (+ 1109us) server_negotiation.cc:407] Connection header received
0924 11:39:02.380882 (+ 2444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:02.380888 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:02.380985 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:02.381103 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:02.382995 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:02.384097 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:02.386091 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:02.386884 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:02.387034 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:02.390604 (+ 3570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:02.390613 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:02.390621 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:39:02.390666 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:02.394319 (+ 3653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:02.395208 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:02.395216 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:02.395222 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:02.395432 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:02.395916 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:02.395921 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:02.395924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:02.396232 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:02.396400 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:02.397129 (+ 729us) server_negotiation.cc:299] Negotiation successful
0924 11:39:02.397373 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":237,"threads_started":1}
I20260924 11:39:02.398886 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:02.377500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:40843 (local address 127.6.212.2:39919)
0924 11:39:02.378247 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:02.378291 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 11:39:02.380516 (+ 2225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:39:02.381405 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:39:02.381417 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:39:02.382224 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:39:02.382812 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:02.382832 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:02.384257 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:02.384263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:02.385944 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:02.385956 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:02.387106 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:02.387111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:02.387305 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:02.388083 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:39:02.388110 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:39:02.390304 (+ 2194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:39:02.394480 (+ 4176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:39:02.394487 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:39:02.394502 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:39:02.395077 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:39:02.395563 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:39:02.395566 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:39:02.395569 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:39:02.395755 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:39:02.396391 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:39:02.396399 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:39:02.396767 (+ 368us) client_negotiation.cc:769] Sending connection context
0924 11:39:02.397058 (+ 291us) client_negotiation.cc:240] Negotiation successful
0924 11:39:02.397621 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":325,"threads_started":1}
I20260924 11:39:02.400537 26693 heartbeater.cc:347] Connected to a master server at 127.6.212.62:40843
I20260924 11:39:02.400918 26693 heartbeater.cc:464] Registering TS with master...
I20260924 11:39:02.401827 26693 heartbeater.cc:511] Master 127.6.212.62:40843 requested a full tablet report, sending...
I20260924 11:39:02.403627 26370 ts_manager.cc:194] Registered new tserver with Master: 7cbccf015b054d999764ce9812020d63 (127.6.212.2:45231)
I20260924 11:39:02.405015 26370 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.2@KRBTEST.COM'} at 127.6.212.2:39919
WARNING: no policy specified for kudu/127.6.212.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab.
Entry for principal kudu/127.6.212.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab.
I20260924 11:39:02.447904 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.3:0
--local_ip_for_outbound_sockets=127.6.212.3
--webserver_interface=127.6.212.3
--webserver_port=0
--tserver_master_addrs=127.6.212.62:40843
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab
--principal=kudu/127.6.212.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:36407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:39:02.792898 26699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:39:02.793179 26699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:39:02.793272 26699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:39:02.801113 26699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:39:02.801234 26699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:39:02.801347 26699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.3
I20260924 11:39:02.811484 26699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:36407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.3.keytab
--principal=kudu/127.6.212.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.212.3
--webserver_port=0
--tserver_master_addrs=127.6.212.62:40843
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.3
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:39:02.812737 26699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:39:02.814087 26699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:39:02.827811 26705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:02.827855 26704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:02.828011 26707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:39:02.828949 26699 server_base.cc:1034] running on GCE node
I20260924 11:39:02.829500 26699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:39:02.830418 26699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:39:02.831619 26699 hybrid_clock.cc:648] HybridClock initialized: now 1790249942831575 us; error 32 us; skew 500 ppm
I20260924 11:39:02.831938 26699 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:39:02 dist-test-slave-cq7c krb5kdc[26325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249942, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:39:02.835322 26699 init.cc:377] Logged in from keytab as kudu/127.6.212.3@KRBTEST.COM (short username kudu)
I20260924 11:39:02.837136 26699 webserver.cc:460] Webserver started at http://127.6.212.3:39403/ using document root <none> and password file <none>
I20260924 11:39:02.838012 26699 fs_manager.cc:359] Metadata directory not provided
I20260924 11:39:02.838126 26699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:39:02.838418 26699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:39:02.841610 26699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-2/data/instance:
uuid: "5c5bc24e356e4bc9b667b913ae69960c"
format_stamp: "Formatted at 2026-09-24 11:39:02 on dist-test-slave-cq7c"
server_key: "a00115e84021ad00600fca9ea36e7145"
server_key_iv: "d543f0c777a45d618bc6ffeac102fee4"
server_key_version: "encryptionkey@0"
I20260924 11:39:02.842403 26699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance:
uuid: "5c5bc24e356e4bc9b667b913ae69960c"
format_stamp: "Formatted at 2026-09-24 11:39:02 on dist-test-slave-cq7c"
server_key: "a00115e84021ad00600fca9ea36e7145"
server_key_iv: "d543f0c777a45d618bc6ffeac102fee4"
server_key_version: "encryptionkey@0"
I20260924 11:39:02.848021 26699 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 11:39:02.851598 26714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:39:02.853137 26699 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:39:02.853336 26699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-2/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-2/wal
uuid: "5c5bc24e356e4bc9b667b913ae69960c"
format_stamp: "Formatted at 2026-09-24 11:39:02 on dist-test-slave-cq7c"
server_key: "a00115e84021ad00600fca9ea36e7145"
server_key_iv: "d543f0c777a45d618bc6ffeac102fee4"
server_key_version: "encryptionkey@0"
I20260924 11:39:02.853547 26699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:39:02.858598 26559 heartbeater.cc:503] Master 127.6.212.62:40843 was elected leader, sending a full tablet report...
I20260924 11:39:02.868852 26699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:39:02.873163 26699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:39:02.873464 26699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:39:02.874545 26699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:39:02.876022 26699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:39:02.876101 26699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:39:02.876271 26699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:39:02.876338 26699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:39:02.910935 26699 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.3:40767
I20260924 11:39:02.910979 26826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.3:40767 every 8 connection(s)
I20260924 11:39:02.912333 26699 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
I20260924 11:39:02.915140 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 26699
I20260924 11:39:02.915401 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance
I20260924 11:39:02.915896 6992 external_mini_cluster.cc:1428] Setting key 8a2b3fc26a0b872a4a25e0b489445b6f
Sep 24 11:39:02 dist-test-slave-cq7c krb5kdc[26325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249942, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.3@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:39:02.934870 26830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:02.916498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:59967 (local address 127.6.212.62:40843)
0924 11:39:02.919894 (+ 3396us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:02.919898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:02.919914 (+ 16us) server_negotiation.cc:407] Connection header received
0924 11:39:02.919956 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:02.919960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:02.920016 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:02.920099 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:02.921708 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:02.922463 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:02.924259 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:02.924667 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:02.924762 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:02.928566 (+ 3804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:02.928571 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:02.928574 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:39:02.928607 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:02.931884 (+ 3277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:02.932596 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:02.932602 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:02.932610 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:02.932725 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:02.933151 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:02.933156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:02.933159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:02.933426 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:02.933590 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:02.934264 (+ 674us) server_negotiation.cc:299] Negotiation successful
0924 11:39:02.934490 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3293,"thread_start_us":3175,"threads_started":1}
I20260924 11:39:02.935930 26829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:02.915903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:40843 (local address 127.6.212.3:59967)
0924 11:39:02.916882 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:02.916944 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 11:39:02.919346 (+ 2402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:39:02.920359 (+ 1013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:39:02.920368 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:39:02.920949 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:39:02.921551 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:02.921569 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:02.922602 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:02.922606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:02.924115 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:02.924127 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:02.924869 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:02.924873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:02.925078 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:02.925837 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:39:02.925861 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:39:02.928374 (+ 2513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:39:02.932021 (+ 3647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:39:02.932029 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:39:02.932045 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:39:02.932448 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:39:02.932843 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:39:02.932848 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:39:02.932852 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:39:02.933033 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:39:02.933553 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:39:02.933560 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:39:02.933904 (+ 344us) client_negotiation.cc:769] Sending connection context
0924 11:39:02.934172 (+ 268us) client_negotiation.cc:240] Negotiation successful
0924 11:39:02.934682 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":378,"threads_started":1}
I20260924 11:39:02.937377 26827 heartbeater.cc:347] Connected to a master server at 127.6.212.62:40843
I20260924 11:39:02.937716 26827 heartbeater.cc:464] Registering TS with master...
I20260924 11:39:02.938452 26827 heartbeater.cc:511] Master 127.6.212.62:40843 requested a full tablet report, sending...
I20260924 11:39:02.939965 26370 ts_manager.cc:194] Registered new tserver with Master: 5c5bc24e356e4bc9b667b913ae69960c (127.6.212.3:40767)
I20260924 11:39:02.940809 26370 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.212.3@KRBTEST.COM'} at 127.6.212.3:59967
I20260924 11:39:02.944013 6992 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:39:02.960660 26830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:02.948136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44404 (local address 127.6.212.62:40843)
0924 11:39:02.948302 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:02.948307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:02.948769 (+ 462us) server_negotiation.cc:407] Connection header received
0924 11:39:02.948909 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:02.948912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:02.948969 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:02.949047 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:02.950442 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:02.951130 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:02.952475 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:02.952865 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:02.952956 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:02.955335 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:02.955340 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:02.955343 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:39:02.955431 (+ 88us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:02.958094 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:02.958862 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:02.958867 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:02.958870 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:02.958997 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:02.959314 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:02.959317 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:02.959319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:02.959689 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:02.959873 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:02.960216 (+ 343us) server_negotiation.cc:299] Negotiation successful
0924 11:39:02.960432 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 11:39:02.964516 26370 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44404:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:39:02.971433 26370 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:39:02.996482 26840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:02.984384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:36489 (local address 127.0.0.1:35082)
0924 11:39:02.986660 (+ 2276us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:02.986707 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 11:39:02.986934 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:39:02.988635 (+ 1701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:39:02.988643 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:39:02.988675 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:39:02.989157 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:02.989176 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:02.991446 (+ 2270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:02.991452 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:02.993530 (+ 2078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:02.993547 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:02.995330 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:02.995363 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:02.995577 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:02.995618 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 11:39:02.995766 (+ 148us) client_negotiation.cc:240] Negotiation successful
0924 11:39:02.995979 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"mutex_wait_us":21,"thread_start_us":784,"threads_started":1}
I20260924 11:39:02.997212 26842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:02.985430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58262 (local address 127.6.212.3:40767)
0924 11:39:02.986222 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:02.986230 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:02.987975 (+ 1745us) server_negotiation.cc:407] Connection header received
0924 11:39:02.988677 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:02.988687 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:02.988942 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:02.989096 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:39:02.990003 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:02.991741 (+ 1738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:02.994316 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:02.995786 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:02.995918 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:02.996155 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:02.996459 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 11:39:02.996803 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":470,"threads_started":1}
I20260924 11:39:02.997362 26843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:02.985351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.3:40767 (local address 127.0.0.1:58262)
0924 11:39:02.987871 (+ 2520us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:02.987887 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:39:02.988494 (+ 607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:39:02.989303 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:39:02.989308 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:39:02.989338 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:39:02.989788 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:02.989801 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:02.991897 (+ 2096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:02.991908 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:02.994151 (+ 2243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:02.994166 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:02.995995 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:02.996001 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:02.996189 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:02.996270 (+ 81us) client_negotiation.cc:769] Sending connection context
0924 11:39:02.996420 (+ 150us) client_negotiation.cc:240] Negotiation successful
0924 11:39:02.996757 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2102,"mutex_wait_us":70,"thread_start_us":1926,"threads_started":1}
I20260924 11:39:02.997807 26845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:02.986710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41860 (local address 127.6.212.2:45231)
0924 11:39:02.988247 (+ 1537us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:02.988262 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:02.988285 (+ 23us) server_negotiation.cc:407] Connection header received
0924 11:39:02.988491 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:02.988497 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:02.988748 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:02.988959 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:39:02.989670 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:02.991128 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:02.994295 (+ 3167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:02.995796 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:02.995922 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:02.996120 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:02.997303 (+ 1183us) server_negotiation.cc:299] Negotiation successful
0924 11:39:02.997481 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1377,"thread_start_us":1238,"threads_started":1}
I20260924 11:39:02.997992 26844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:02.986446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.2:45231 (local address 127.0.0.1:41860)
0924 11:39:02.988216 (+ 1770us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:02.988232 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:39:02.988328 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:39:02.988967 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:39:02.988974 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:39:02.988999 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:39:02.989489 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:02.989500 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:02.991252 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:02.991260 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:02.993810 (+ 2550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:02.993827 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:02.996393 (+ 2566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:02.996401 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:02.996584 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:02.996613 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 11:39:02.996695 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 11:39:02.996817 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1673,"mutex_wait_us":27,"thread_start_us":1336,"threads_started":1}
I20260924 11:39:02.998044 26841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:02.984891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35082 (local address 127.6.212.1:36489)
0924 11:39:02.985616 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:02.985626 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:02.987968 (+ 2342us) server_negotiation.cc:407] Connection header received
0924 11:39:02.988134 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:02.988142 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:02.988411 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:02.988610 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:39:02.989386 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:02.991248 (+ 1862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:02.993776 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:02.995075 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:02.997028 (+ 1953us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:02.997349 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:02.997435 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 11:39:02.997639 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":719,"threads_started":1}
I20260924 11:39:03.001469 26628 tablet_service.cc:1467] Processing CreateTablet for tablet 630d3cfb42e14016a5233895927d0f2a (DEFAULT_TABLE table=test-table [id=bcefc1c8f5e34afbaa00cf0f90bdac91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:39:03.001570 26762 tablet_service.cc:1467] Processing CreateTablet for tablet 630d3cfb42e14016a5233895927d0f2a (DEFAULT_TABLE table=test-table [id=bcefc1c8f5e34afbaa00cf0f90bdac91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:39:03.001570 26494 tablet_service.cc:1467] Processing CreateTablet for tablet 630d3cfb42e14016a5233895927d0f2a (DEFAULT_TABLE table=test-table [id=bcefc1c8f5e34afbaa00cf0f90bdac91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:39:03.002825 26628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 630d3cfb42e14016a5233895927d0f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:39:03.002867 26494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 630d3cfb42e14016a5233895927d0f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:39:03.002898 26762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 630d3cfb42e14016a5233895927d0f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:39:03.014784 26846 tablet_bootstrap.cc:492] T 630d3cfb42e14016a5233895927d0f2a P 7cbccf015b054d999764ce9812020d63: Bootstrap starting.
I20260924 11:39:03.018584 26846 tablet_bootstrap.cc:654] T 630d3cfb42e14016a5233895927d0f2a P 7cbccf015b054d999764ce9812020d63: Neither blocks nor log segments found. Creating new log.
I20260924 11:39:03.019974 26846 log.cc:824] T 630d3cfb42e14016a5233895927d0f2a P 7cbccf015b054d999764ce9812020d63: Log is configured to *not* fsync() on all Append() calls
I20260924 11:39:03.020955 26848 tablet_bootstrap.cc:492] T 630d3cfb42e14016a5233895927d0f2a P dfea1e60e88d440c828958db825fa574: Bootstrap starting.
I20260924 11:39:03.021606 26847 tablet_bootstrap.cc:492] T 630d3cfb42e14016a5233895927d0f2a P 5c5bc24e356e4bc9b667b913ae69960c: Bootstrap starting.
I20260924 11:39:03.023901 26846 tablet_bootstrap.cc:492] T 630d3cfb42e14016a5233895927d0f2a P 7cbccf015b054d999764ce9812020d63: No bootstrap required, opened a new log
I20260924 11:39:03.024175 26846 ts_tablet_manager.cc:1397] T 630d3cfb42e14016a5233895927d0f2a P 7cbccf015b054d999764ce9812020d63: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260924 11:39:03.025292 26848 tablet_bootstrap.cc:654] T 630d3cfb42e14016a5233895927d0f2a P dfea1e60e88d440c828958db825fa574: Neither blocks nor log segments found. Creating new log.
I20260924 11:39:03.025399 26847 tablet_bootstrap.cc:654] T 630d3cfb42e14016a5233895927d0f2a P 5c5bc24e356e4bc9b667b913ae69960c: Neither blocks nor log segments found. Creating new log.
I20260924 11:39:03.026557 26848 log.cc:824] T 630d3cfb42e14016a5233895927d0f2a P dfea1e60e88d440c828958db825fa574: Log is configured to *not* fsync() on all Append() calls
I20260924 11:39:03.026687 26847 log.cc:824] T 630d3cfb42e14016a5233895927d0f2a P 5c5bc24e356e4bc9b667b913ae69960c: Log is configured to *not* fsync() on all Append() calls
I20260924 11:39:03.029881 26848 tablet_bootstrap.cc:492] T 630d3cfb42e14016a5233895927d0f2a P dfea1e60e88d440c828958db825fa574: No bootstrap required, opened a new log
I20260924 11:39:03.029881 26847 tablet_bootstrap.cc:492] T 630d3cfb42e14016a5233895927d0f2a P 5c5bc24e356e4bc9b667b913ae69960c: No bootstrap required, opened a new log
I20260924 11:39:03.030171 26848 ts_tablet_manager.cc:1397] T 630d3cfb42e14016a5233895927d0f2a P dfea1e60e88d440c828958db825fa574: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20260924 11:39:03.030172 26847 ts_tablet_manager.cc:1397] T 630d3cfb42e14016a5233895927d0f2a P 5c5bc24e356e4bc9b667b913ae69960c: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260924 11:39:03.032836 26846 raft_consensus.cc:348] T 630d3cfb42e14016a5233895927d0f2a P 7cbccf015b054d999764ce9812020d63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfea1e60e88d440c828958db825fa574" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 36489 } } peers { permanent_uuid: "7cbccf015b054d999764ce9812020d63" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45231 } } peers { permanent_uuid: "5c5bc24e356e4bc9b667b913ae69960c" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 40767 } }
I20260924 11:39:03.033234 26846 raft_consensus.cc:374] T 630d3cfb42e14016a5233895927d0f2a P 7cbccf015b054d999764ce9812020d63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:39:03.033336 26846 raft_consensus.cc:729] T 630d3cfb42e14016a5233895927d0f2a P 7cbccf015b054d999764ce9812020d63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cbccf015b054d999764ce9812020d63, State: Initialized, Role: FOLLOWER
I20260924 11:39:03.033955 26846 consensus_queue.cc:260] T 630d3cfb42e14016a5233895927d0f2a P 7cbccf015b054d999764ce9812020d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfea1e60e88d440c828958db825fa574" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 36489 } } peers { permanent_uuid: "7cbccf015b054d999764ce9812020d63" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45231 } } peers { permanent_uuid: "5c5bc24e356e4bc9b667b913ae69960c" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 40767 } }
I20260924 11:39:03.035091 26693 heartbeater.cc:503] Master 127.6.212.62:40843 was elected leader, sending a full tablet report...
I20260924 11:39:03.035897 26846 ts_tablet_manager.cc:1428] T 630d3cfb42e14016a5233895927d0f2a P 7cbccf015b054d999764ce9812020d63: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20260924 11:39:03.042464 26848 raft_consensus.cc:348] T 630d3cfb42e14016a5233895927d0f2a P dfea1e60e88d440c828958db825fa574 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfea1e60e88d440c828958db825fa574" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 36489 } } peers { permanent_uuid: "7cbccf015b054d999764ce9812020d63" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45231 } } peers { permanent_uuid: "5c5bc24e356e4bc9b667b913ae69960c" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 40767 } }
I20260924 11:39:03.042897 26848 raft_consensus.cc:374] T 630d3cfb42e14016a5233895927d0f2a P dfea1e60e88d440c828958db825fa574 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:39:03.043030 26848 raft_consensus.cc:729] T 630d3cfb42e14016a5233895927d0f2a P dfea1e60e88d440c828958db825fa574 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfea1e60e88d440c828958db825fa574, State: Initialized, Role: FOLLOWER
I20260924 11:39:03.043620 26847 raft_consensus.cc:348] T 630d3cfb42e14016a5233895927d0f2a P 5c5bc24e356e4bc9b667b913ae69960c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfea1e60e88d440c828958db825fa574" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 36489 } } peers { permanent_uuid: "7cbccf015b054d999764ce9812020d63" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45231 } } peers { permanent_uuid: "5c5bc24e356e4bc9b667b913ae69960c" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 40767 } }
I20260924 11:39:03.043708 26848 consensus_queue.cc:260] T 630d3cfb42e14016a5233895927d0f2a P dfea1e60e88d440c828958db825fa574 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfea1e60e88d440c828958db825fa574" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 36489 } } peers { permanent_uuid: "7cbccf015b054d999764ce9812020d63" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45231 } } peers { permanent_uuid: "5c5bc24e356e4bc9b667b913ae69960c" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 40767 } }
I20260924 11:39:03.043982 26847 raft_consensus.cc:374] T 630d3cfb42e14016a5233895927d0f2a P 5c5bc24e356e4bc9b667b913ae69960c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:39:03.044103 26847 raft_consensus.cc:729] T 630d3cfb42e14016a5233895927d0f2a P 5c5bc24e356e4bc9b667b913ae69960c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c5bc24e356e4bc9b667b913ae69960c, State: Initialized, Role: FOLLOWER
I20260924 11:39:03.044721 26847 consensus_queue.cc:260] T 630d3cfb42e14016a5233895927d0f2a P 5c5bc24e356e4bc9b667b913ae69960c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfea1e60e88d440c828958db825fa574" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 36489 } } peers { permanent_uuid: "7cbccf015b054d999764ce9812020d63" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45231 } } peers { permanent_uuid: "5c5bc24e356e4bc9b667b913ae69960c" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 40767 } }
I20260924 11:39:03.045749 26848 ts_tablet_manager.cc:1428] T 630d3cfb42e14016a5233895927d0f2a P dfea1e60e88d440c828958db825fa574: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260924 11:39:03.045944 26827 heartbeater.cc:503] Master 127.6.212.62:40843 was elected leader, sending a full tablet report...
I20260924 11:39:03.046532 26847 ts_tablet_manager.cc:1428] T 630d3cfb42e14016a5233895927d0f2a P 5c5bc24e356e4bc9b667b913ae69960c: Time spent starting tablet: real 0.016s user 0.015s sys 0.001s
W20260924 11:39:03.073568 26560 tablet.cc:2369] T 630d3cfb42e14016a5233895927d0f2a P dfea1e60e88d440c828958db825fa574: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:39:03.126050 26694 tablet.cc:2369] T 630d3cfb42e14016a5233895927d0f2a P 7cbccf015b054d999764ce9812020d63: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:39:03.164094 26828 tablet.cc:2369] T 630d3cfb42e14016a5233895927d0f2a P 5c5bc24e356e4bc9b667b913ae69960c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:39:03.353823 26854 raft_consensus.cc:482] T 630d3cfb42e14016a5233895927d0f2a P 5c5bc24e356e4bc9b667b913ae69960c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:39:03.354122 26854 raft_consensus.cc:504] T 630d3cfb42e14016a5233895927d0f2a P 5c5bc24e356e4bc9b667b913ae69960c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfea1e60e88d440c828958db825fa574" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 36489 } } peers { permanent_uuid: "7cbccf015b054d999764ce9812020d63" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45231 } } peers { permanent_uuid: "5c5bc24e356e4bc9b667b913ae69960c" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 40767 } }
I20260924 11:39:03.355821 26854 leader_election.cc:288] T 630d3cfb42e14016a5233895927d0f2a P 5c5bc24e356e4bc9b667b913ae69960c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfea1e60e88d440c828958db825fa574 (127.6.212.1:36489), 7cbccf015b054d999764ce9812020d63 (127.6.212.2:45231)
I20260924 11:39:03.362211 26841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:03.356204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:56053 (local address 127.6.212.1:36489)
0924 11:39:03.356417 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:03.356423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:03.356446 (+ 23us) server_negotiation.cc:407] Connection header received
0924 11:39:03.356570 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:03.356577 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:03.356656 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:03.356766 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:39:03.357399 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:03.358453 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:03.360313 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:03.361432 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:03.361533 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:03.361604 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:03.361946 (+ 342us) server_negotiation.cc:299] Negotiation successful
0924 11:39:03.362034 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 11:39:03.362335 26829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:03.356091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:36489 (local address 127.6.212.3:56053)
0924 11:39:03.356288 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:03.356310 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 11:39:03.356429 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:39:03.356947 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:39:03.356953 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:39:03.356981 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:39:03.357221 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:03.357235 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:03.358580 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:03.358585 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:03.360150 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:03.360161 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:03.361582 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:03.361588 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:03.361774 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:03.361814 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 11:39:03.361913 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 11:39:03.362046 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 11:39:03.363236 26845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:03.356724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:38093 (local address 127.6.212.2:45231)
0924 11:39:03.357053 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:03.357059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:03.357198 (+ 139us) server_negotiation.cc:407] Connection header received
0924 11:39:03.357311 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:03.357318 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:03.357400 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:03.357466 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:39:03.358056 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:03.359111 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:03.361163 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:03.362210 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:03.362339 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:03.362436 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:03.362978 (+ 542us) server_negotiation.cc:299] Negotiation successful
0924 11:39:03.363070 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193}
I20260924 11:39:03.363401 26855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:03.356487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.2:45231 (local address 127.6.212.3:38093)
0924 11:39:03.357105 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:03.357119 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 11:39:03.357214 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:39:03.357512 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:39:03.357517 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:39:03.357542 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:39:03.357908 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:03.357920 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:03.359297 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:03.359301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:03.361001 (+ 1700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:03.361014 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:03.362584 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:03.362591 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:03.362782 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:03.362834 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 11:39:03.362932 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 11:39:03.363061 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":352,"threads_started":1}
I20260924 11:39:03.363415 26514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "630d3cfb42e14016a5233895927d0f2a" candidate_uuid: "5c5bc24e356e4bc9b667b913ae69960c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfea1e60e88d440c828958db825fa574" is_pre_election: true
I20260924 11:39:03.364085 26514 raft_consensus.cc:2454] T 630d3cfb42e14016a5233895927d0f2a P dfea1e60e88d440c828958db825fa574 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c5bc24e356e4bc9b667b913ae69960c in term 0.
I20260924 11:39:03.364444 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "630d3cfb42e14016a5233895927d0f2a" candidate_uuid: "5c5bc24e356e4bc9b667b913ae69960c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cbccf015b054d999764ce9812020d63" is_pre_election: true
I20260924 11:39:03.364805 26717 leader_election.cc:302] T 630d3cfb42e14016a5233895927d0f2a P 5c5bc24e356e4bc9b667b913ae69960c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c5bc24e356e4bc9b667b913ae69960c, dfea1e60e88d440c828958db825fa574; no voters:
I20260924 11:39:03.364981 26648 raft_consensus.cc:2454] T 630d3cfb42e14016a5233895927d0f2a P 7cbccf015b054d999764ce9812020d63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c5bc24e356e4bc9b667b913ae69960c in term 0.
I20260924 11:39:03.365239 26854 raft_consensus.cc:2781] T 630d3cfb42e14016a5233895927d0f2a P 5c5bc24e356e4bc9b667b913ae69960c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:39:03.365399 26854 raft_consensus.cc:482] T 630d3cfb42e14016a5233895927d0f2a P 5c5bc24e356e4bc9b667b913ae69960c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:39:03.365523 26854 raft_consensus.cc:3037] T 630d3cfb42e14016a5233895927d0f2a P 5c5bc24e356e4bc9b667b913ae69960c [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:39:03.367923 26854 raft_consensus.cc:504] T 630d3cfb42e14016a5233895927d0f2a P 5c5bc24e356e4bc9b667b913ae69960c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfea1e60e88d440c828958db825fa574" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 36489 } } peers { permanent_uuid: "7cbccf015b054d999764ce9812020d63" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45231 } } peers { permanent_uuid: "5c5bc24e356e4bc9b667b913ae69960c" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 40767 } }
I20260924 11:39:03.368486 26854 leader_election.cc:288] T 630d3cfb42e14016a5233895927d0f2a P 5c5bc24e356e4bc9b667b913ae69960c [CANDIDATE]: Term 1 election: Requested vote from peers dfea1e60e88d440c828958db825fa574 (127.6.212.1:36489), 7cbccf015b054d999764ce9812020d63 (127.6.212.2:45231)
I20260924 11:39:03.369014 26514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "630d3cfb42e14016a5233895927d0f2a" candidate_uuid: "5c5bc24e356e4bc9b667b913ae69960c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfea1e60e88d440c828958db825fa574"
I20260924 11:39:03.369130 26648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "630d3cfb42e14016a5233895927d0f2a" candidate_uuid: "5c5bc24e356e4bc9b667b913ae69960c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cbccf015b054d999764ce9812020d63"
I20260924 11:39:03.369262 26514 raft_consensus.cc:3037] T 630d3cfb42e14016a5233895927d0f2a P dfea1e60e88d440c828958db825fa574 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:39:03.369393 26648 raft_consensus.cc:3037] T 630d3cfb42e14016a5233895927d0f2a P 7cbccf015b054d999764ce9812020d63 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:39:03.371635 26514 raft_consensus.cc:2454] T 630d3cfb42e14016a5233895927d0f2a P dfea1e60e88d440c828958db825fa574 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c5bc24e356e4bc9b667b913ae69960c in term 1.
I20260924 11:39:03.371685 26648 raft_consensus.cc:2454] T 630d3cfb42e14016a5233895927d0f2a P 7cbccf015b054d999764ce9812020d63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c5bc24e356e4bc9b667b913ae69960c in term 1.
I20260924 11:39:03.372095 26717 leader_election.cc:302] T 630d3cfb42e14016a5233895927d0f2a P 5c5bc24e356e4bc9b667b913ae69960c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c5bc24e356e4bc9b667b913ae69960c, dfea1e60e88d440c828958db825fa574; no voters:
I20260924 11:39:03.372408 26854 raft_consensus.cc:2781] T 630d3cfb42e14016a5233895927d0f2a P 5c5bc24e356e4bc9b667b913ae69960c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:39:03.372877 26854 raft_consensus.cc:686] T 630d3cfb42e14016a5233895927d0f2a P 5c5bc24e356e4bc9b667b913ae69960c [term 1 LEADER]: Becoming Leader. State: Replica: 5c5bc24e356e4bc9b667b913ae69960c, State: Running, Role: LEADER
I20260924 11:39:03.373428 26854 consensus_queue.cc:237] T 630d3cfb42e14016a5233895927d0f2a P 5c5bc24e356e4bc9b667b913ae69960c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfea1e60e88d440c828958db825fa574" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 36489 } } peers { permanent_uuid: "7cbccf015b054d999764ce9812020d63" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45231 } } peers { permanent_uuid: "5c5bc24e356e4bc9b667b913ae69960c" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 40767 } }
I20260924 11:39:03.377979 26370 catalog_manager.cc:5495] T 630d3cfb42e14016a5233895927d0f2a P 5c5bc24e356e4bc9b667b913ae69960c reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c5bc24e356e4bc9b667b913ae69960c (127.6.212.3). New cstate: current_term: 1 leader_uuid: "5c5bc24e356e4bc9b667b913ae69960c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfea1e60e88d440c828958db825fa574" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 36489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7cbccf015b054d999764ce9812020d63" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c5bc24e356e4bc9b667b913ae69960c" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 40767 } health_report { overall_health: HEALTHY } } }
I20260924 11:39:03.453190 26842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:03.447307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58266 (local address 127.6.212.3:40767)
0924 11:39:03.447572 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:03.447579 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:03.447599 (+ 20us) server_negotiation.cc:407] Connection header received
0924 11:39:03.447828 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:03.447833 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:03.447916 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:03.448086 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:39:03.448485 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:03.449293 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:03.450806 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:03.451279 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:03.451458 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:03.452010 (+ 552us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:39:03.452544 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:39:03.452730 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:03.452804 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 11:39:03.452890 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260924 11:39:03.462338 26514 raft_consensus.cc:1261] T 630d3cfb42e14016a5233895927d0f2a P dfea1e60e88d440c828958db825fa574 [term 1 FOLLOWER]: Refusing update from remote peer 5c5bc24e356e4bc9b667b913ae69960c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:39:03.462340 26648 raft_consensus.cc:1261] T 630d3cfb42e14016a5233895927d0f2a P 7cbccf015b054d999764ce9812020d63 [term 1 FOLLOWER]: Refusing update from remote peer 5c5bc24e356e4bc9b667b913ae69960c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:39:03.463471 26856 consensus_queue.cc:1035] T 630d3cfb42e14016a5233895927d0f2a P 5c5bc24e356e4bc9b667b913ae69960c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfea1e60e88d440c828958db825fa574" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:39:03.463873 26854 consensus_queue.cc:1035] T 630d3cfb42e14016a5233895927d0f2a P 5c5bc24e356e4bc9b667b913ae69960c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cbccf015b054d999764ce9812020d63" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 45231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:39:03.476738 26861 mvcc.cc:205] Tried to move back new op lower bound from 7332863768410251264 to 7332863768062144512. Current Snapshot: MvccSnapshot[applied={T|T < 7332863768410251264}]
I20260924 11:39:03.490447 26370 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:39:03.490731 26370 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:39:03.494513 26370 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e58765b92d6e4bcf9d3673146b0e4345: Sending DeleteTablet for 3 replicas of tablet 630d3cfb42e14016a5233895927d0f2a
I20260924 11:39:03.495760 26762 tablet_service.cc:1514] Processing DeleteTablet for tablet 630d3cfb42e14016a5233895927d0f2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:39:03 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:58262
I20260924 11:39:03.495796 26494 tablet_service.cc:1514] Processing DeleteTablet for tablet 630d3cfb42e14016a5233895927d0f2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:39:03 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:35082
I20260924 11:39:03.496547 26628 tablet_service.cc:1514] Processing DeleteTablet for tablet 630d3cfb42e14016a5233895927d0f2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:39:03 UTC) from {username='kudu', principal='kudu/127.6.212.62@KRBTEST.COM'} at 127.0.0.1:41860
I20260924 11:39:03.496845 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 26431
I20260924 11:39:03.500639 26871 tablet_replica.cc:330] T 630d3cfb42e14016a5233895927d0f2a P 7cbccf015b054d999764ce9812020d63: stopping tablet replica
I20260924 11:39:03.501204 26871 raft_consensus.cc:2229] T 630d3cfb42e14016a5233895927d0f2a P 7cbccf015b054d999764ce9812020d63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:39:03.501860 26871 raft_consensus.cc:2258] T 630d3cfb42e14016a5233895927d0f2a P 7cbccf015b054d999764ce9812020d63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:39:03.504715 26871 ts_tablet_manager.cc:1905] T 630d3cfb42e14016a5233895927d0f2a P 7cbccf015b054d999764ce9812020d63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:39:03.514431 26871 ts_tablet_manager.cc:1918] T 630d3cfb42e14016a5233895927d0f2a P 7cbccf015b054d999764ce9812020d63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:39:03.514638 26871 log.cc:1196] T 630d3cfb42e14016a5233895927d0f2a P 7cbccf015b054d999764ce9812020d63: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-1/wal/wals/630d3cfb42e14016a5233895927d0f2a
I20260924 11:39:03.515305 26871 ts_tablet_manager.cc:1939] T 630d3cfb42e14016a5233895927d0f2a P 7cbccf015b054d999764ce9812020d63: Deleting consensus metadata
I20260924 11:39:03.517742 26356 catalog_manager.cc:4841] TS 7cbccf015b054d999764ce9812020d63 (127.6.212.2:45231): tablet 630d3cfb42e14016a5233895927d0f2a (table test-table [id=bcefc1c8f5e34afbaa00cf0f90bdac91]) successfully deleted
I20260924 11:39:03.518818 26869 tablet_replica.cc:330] T 630d3cfb42e14016a5233895927d0f2a P 5c5bc24e356e4bc9b667b913ae69960c: stopping tablet replica
I20260924 11:39:03.519412 26869 raft_consensus.cc:2229] T 630d3cfb42e14016a5233895927d0f2a P 5c5bc24e356e4bc9b667b913ae69960c [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:39:03.520227 26869 raft_consensus.cc:2258] T 630d3cfb42e14016a5233895927d0f2a P 5c5bc24e356e4bc9b667b913ae69960c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:39:03.521132 26870 tablet_replica.cc:330] T 630d3cfb42e14016a5233895927d0f2a P dfea1e60e88d440c828958db825fa574: stopping tablet replica
I20260924 11:39:03.521586 26870 raft_consensus.cc:2229] T 630d3cfb42e14016a5233895927d0f2a P dfea1e60e88d440c828958db825fa574 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:39:03.522207 26870 raft_consensus.cc:2258] T 630d3cfb42e14016a5233895927d0f2a P dfea1e60e88d440c828958db825fa574 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:39:03.522998 26869 ts_tablet_manager.cc:1905] T 630d3cfb42e14016a5233895927d0f2a P 5c5bc24e356e4bc9b667b913ae69960c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:39:03.524470 26870 ts_tablet_manager.cc:1905] T 630d3cfb42e14016a5233895927d0f2a P dfea1e60e88d440c828958db825fa574: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 11:39:03 dist-test-slave-cq7c krb5kdc[26325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.1@KRBTEST.COM
I20260924 11:39:03.530647 26869 ts_tablet_manager.cc:1918] T 630d3cfb42e14016a5233895927d0f2a P 5c5bc24e356e4bc9b667b913ae69960c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:39:03.530839 26869 log.cc:1196] T 630d3cfb42e14016a5233895927d0f2a P 5c5bc24e356e4bc9b667b913ae69960c: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-2/wal/wals/630d3cfb42e14016a5233895927d0f2a
I20260924 11:39:03.531383 26869 ts_tablet_manager.cc:1939] T 630d3cfb42e14016a5233895927d0f2a P 5c5bc24e356e4bc9b667b913ae69960c: Deleting consensus metadata
I20260924 11:39:03.532632 26870 ts_tablet_manager.cc:1918] T 630d3cfb42e14016a5233895927d0f2a P dfea1e60e88d440c828958db825fa574: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:39:03.532836 26870 log.cc:1196] T 630d3cfb42e14016a5233895927d0f2a P dfea1e60e88d440c828958db825fa574: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790249617057452-6992-0/minicluster-data/ts-0/wal/wals/630d3cfb42e14016a5233895927d0f2a
I20260924 11:39:03.533269 26355 catalog_manager.cc:4841] TS 5c5bc24e356e4bc9b667b913ae69960c (127.6.212.3:40767): tablet 630d3cfb42e14016a5233895927d0f2a (table test-table [id=bcefc1c8f5e34afbaa00cf0f90bdac91]) successfully deleted
I20260924 11:39:03.533450 26870 ts_tablet_manager.cc:1939] T 630d3cfb42e14016a5233895927d0f2a P dfea1e60e88d440c828958db825fa574: Deleting consensus metadata
I20260924 11:39:03.535579 26356 catalog_manager.cc:4841] TS dfea1e60e88d440c828958db825fa574 (127.6.212.1:36489): tablet 630d3cfb42e14016a5233895927d0f2a (table test-table [id=bcefc1c8f5e34afbaa00cf0f90bdac91]) successfully deleted
I20260924 11:39:03.539254 26841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:03.498129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35088 (local address 127.6.212.1:36489)
0924 11:39:03.498371 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:03.498378 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:03.518066 (+ 19688us) server_negotiation.cc:407] Connection header received
0924 11:39:03.518289 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:03.518296 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:03.518409 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:03.518527 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:03.520723 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:03.522019 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:03.524112 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:03.524826 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:03.524945 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:03.531129 (+ 6184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:03.531144 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:03.531165 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:39:03.531227 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:03.535954 (+ 4727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:03.536918 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:03.536925 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:03.536927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:03.537064 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:03.537482 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:03.537487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:03.537490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:03.538015 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:03.538243 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:03.538662 (+ 419us) server_negotiation.cc:299] Negotiation successful
0924 11:39:03.538837 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 11:39:03.540052 26554 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:39:03.631748 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 26431
I20260924 11:39:03.650048 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 26565
Sep 24 11:39:03 dist-test-slave-cq7c krb5kdc[26325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.2@KRBTEST.COM
I20260924 11:39:03.666721 26845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:03.650901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41866 (local address 127.6.212.2:45231)
0924 11:39:03.651075 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:03.651080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:03.651102 (+ 22us) server_negotiation.cc:407] Connection header received
0924 11:39:03.651172 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:03.651177 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:03.651259 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:03.651410 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:03.652672 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:03.653797 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:03.655004 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:03.655778 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:03.655910 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:03.660213 (+ 4303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:03.660233 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:03.660258 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:39:03.660299 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:03.663880 (+ 3581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:03.664555 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:03.664574 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:03.664577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:03.664740 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:03.665080 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:03.665084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:03.665088 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:03.665555 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:03.665875 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:03.666119 (+ 244us) server_negotiation.cc:299] Negotiation successful
0924 11:39:03.666345 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 11:39:03.667459 26688 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:39:03.762027 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 26565
I20260924 11:39:03.779831 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 26699
Sep 24 11:39:03 dist-test-slave-cq7c krb5kdc[26325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.3@KRBTEST.COM
I20260924 11:39:03.796056 26842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:03.780665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58282 (local address 127.6.212.3:40767)
0924 11:39:03.780842 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:03.780849 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:03.780873 (+ 24us) server_negotiation.cc:407] Connection header received
0924 11:39:03.781096 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:03.781102 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:03.781219 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:03.781373 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:03.782700 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:03.783572 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:03.784940 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:03.785452 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:03.785602 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:03.789903 (+ 4301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:03.789915 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:03.789930 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:39:03.789963 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:03.793183 (+ 3220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:03.793896 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:03.793906 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:03.793909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:03.794068 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:03.794444 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:03.794451 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:03.794454 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:03.794813 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:03.795155 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:03.795430 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 11:39:03.795680 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 11:39:03.796751 26822 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:39:03.922415 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 26699
I20260924 11:39:03.942585 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 26338
I20260924 11:39:03.943769 26400 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:39:04.056228 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 26338
2026-09-24T11:39:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4822 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T11:39:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:39:04Z Disabled control of system clock
I20260924 11:39:04.138321 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:35771
--webserver_interface=127.6.212.62
--webserver_port=0
--builtin_ntp_servers=127.6.212.20:45907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:35771
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 11:39:04.476996 26880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:39:04.477313 26880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:39:04.477396 26880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:39:04.485251 26880 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:39:04.485379 26880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:39:04.485420 26880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:39:04.485471 26880 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:39:04.485515 26880 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:39:04.495025 26880 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:45907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:35771
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:35771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.212.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:39:04.496424 26880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:39:04.498256 26880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:39:04.510524 26888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:04.510601 26886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:04.510568 26885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:39:04.511488 26880 server_base.cc:1034] running on GCE node
I20260924 11:39:04.512311 26880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:39:04.513834 26880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:39:04.515089 26880 hybrid_clock.cc:648] HybridClock initialized: now 1790249944515036 us; error 43 us; skew 500 ppm
I20260924 11:39:04.515520 26880 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:39:04.517997 26880 webserver.cc:460] Webserver started at http://127.6.212.62:33427/ using document root <none> and password file <none>
I20260924 11:39:04.518725 26880 fs_manager.cc:359] Metadata directory not provided
I20260924 11:39:04.518862 26880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:39:04.519189 26880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:39:04.522226 26880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "3d8a9ac84eaa4d0c954c8a87dcbc82bb"
format_stamp: "Formatted at 2026-09-24 11:39:04 on dist-test-slave-cq7c"
server_key: "2b5688acd017ebe8f70e6765574e146a"
server_key_iv: "b65b8e65d8c80e1f5a5caf50310b67b6"
server_key_version: "encryptionkey@0"
I20260924 11:39:04.523061 26880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "3d8a9ac84eaa4d0c954c8a87dcbc82bb"
format_stamp: "Formatted at 2026-09-24 11:39:04 on dist-test-slave-cq7c"
server_key: "2b5688acd017ebe8f70e6765574e146a"
server_key_iv: "b65b8e65d8c80e1f5a5caf50310b67b6"
server_key_version: "encryptionkey@0"
I20260924 11:39:04.528373 26880 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 11:39:04.532079 26894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:39:04.533651 26880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:39:04.533851 26880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "3d8a9ac84eaa4d0c954c8a87dcbc82bb"
format_stamp: "Formatted at 2026-09-24 11:39:04 on dist-test-slave-cq7c"
server_key: "2b5688acd017ebe8f70e6765574e146a"
server_key_iv: "b65b8e65d8c80e1f5a5caf50310b67b6"
server_key_version: "encryptionkey@0"
I20260924 11:39:04.534078 26880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:39:04.561606 26880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:39:04.562549 26880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:39:04.562872 26880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:39:04.583186 26880 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.62:35771
I20260924 11:39:04.583207 26945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.62:35771 every 8 connection(s)
I20260924 11:39:04.584791 26880 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:39:04.588608 26946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:39:04.594806 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 26880
I20260924 11:39:04.595013 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:39:04.595546 6992 external_mini_cluster.cc:1428] Setting key 017ca286fa3dc1c2dd244d4f7d643e40
I20260924 11:39:04.599519 26946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d8a9ac84eaa4d0c954c8a87dcbc82bb: Bootstrap starting.
I20260924 11:39:04.602602 26946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d8a9ac84eaa4d0c954c8a87dcbc82bb: Neither blocks nor log segments found. Creating new log.
I20260924 11:39:04.603718 26946 log.cc:824] T 00000000000000000000000000000000 P 3d8a9ac84eaa4d0c954c8a87dcbc82bb: Log is configured to *not* fsync() on all Append() calls
I20260924 11:39:04.606583 26946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d8a9ac84eaa4d0c954c8a87dcbc82bb: No bootstrap required, opened a new log
I20260924 11:39:04.610476 26949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:04.597437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33200 (local address 127.6.212.62:35771)
0924 11:39:04.598416 (+ 979us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:04.598428 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:04.598470 (+ 42us) server_negotiation.cc:407] Connection header received
0924 11:39:04.600427 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:04.600450 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:04.600946 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:04.601355 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:04.601952 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:04.603478 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:04.605576 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:04.606327 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:04.606545 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:04.607851 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:04.607860 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:04.607874 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:39:04.608473 (+ 599us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:04.608560 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:39:04.608598 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:04.608769 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:04.609046 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 11:39:04.609421 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":300,"threads_started":1}
I20260924 11:39:04.615093 26946 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d8a9ac84eaa4d0c954c8a87dcbc82bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8a9ac84eaa4d0c954c8a87dcbc82bb" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 35771 } }
I20260924 11:39:04.615487 26946 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d8a9ac84eaa4d0c954c8a87dcbc82bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:39:04.615608 26946 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d8a9ac84eaa4d0c954c8a87dcbc82bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d8a9ac84eaa4d0c954c8a87dcbc82bb, State: Initialized, Role: FOLLOWER
I20260924 11:39:04.616137 26946 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d8a9ac84eaa4d0c954c8a87dcbc82bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8a9ac84eaa4d0c954c8a87dcbc82bb" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 35771 } }
I20260924 11:39:04.616410 26946 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d8a9ac84eaa4d0c954c8a87dcbc82bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:39:04.616552 26946 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d8a9ac84eaa4d0c954c8a87dcbc82bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:39:04.616729 26946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d8a9ac84eaa4d0c954c8a87dcbc82bb [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:39:04.619089 26946 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d8a9ac84eaa4d0c954c8a87dcbc82bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8a9ac84eaa4d0c954c8a87dcbc82bb" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 35771 } }
I20260924 11:39:04.619653 26946 leader_election.cc:302] T 00000000000000000000000000000000 P 3d8a9ac84eaa4d0c954c8a87dcbc82bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d8a9ac84eaa4d0c954c8a87dcbc82bb; no voters:
I20260924 11:39:04.620237 26946 leader_election.cc:288] T 00000000000000000000000000000000 P 3d8a9ac84eaa4d0c954c8a87dcbc82bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:39:04.620404 26951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d8a9ac84eaa4d0c954c8a87dcbc82bb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:39:04.621189 26951 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d8a9ac84eaa4d0c954c8a87dcbc82bb [term 1 LEADER]: Becoming Leader. State: Replica: 3d8a9ac84eaa4d0c954c8a87dcbc82bb, State: Running, Role: LEADER
I20260924 11:39:04.621744 26951 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d8a9ac84eaa4d0c954c8a87dcbc82bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8a9ac84eaa4d0c954c8a87dcbc82bb" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 35771 } }
I20260924 11:39:04.622421 26946 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d8a9ac84eaa4d0c954c8a87dcbc82bb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:39:04.624495 26952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d8a9ac84eaa4d0c954c8a87dcbc82bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d8a9ac84eaa4d0c954c8a87dcbc82bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8a9ac84eaa4d0c954c8a87dcbc82bb" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 35771 } } }
I20260924 11:39:04.624518 26953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d8a9ac84eaa4d0c954c8a87dcbc82bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d8a9ac84eaa4d0c954c8a87dcbc82bb. Latest consensus state: current_term: 1 leader_uuid: "3d8a9ac84eaa4d0c954c8a87dcbc82bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8a9ac84eaa4d0c954c8a87dcbc82bb" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 35771 } } }
I20260924 11:39:04.624959 26953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d8a9ac84eaa4d0c954c8a87dcbc82bb [sys.catalog]: This master's current role is: LEADER
I20260924 11:39:04.624959 26952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d8a9ac84eaa4d0c954c8a87dcbc82bb [sys.catalog]: This master's current role is: LEADER
I20260924 11:39:04.629316 26957 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:39:04.634312 26957 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:39:04.643953 26957 catalog_manager.cc:1347] Generated new cluster ID: 268331cf26aa4630ad2137749df279f5
I20260924 11:39:04.644096 26957 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:39:04.662271 26957 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:39:04.663723 26957 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:39:04.671293 26957 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d8a9ac84eaa4d0c954c8a87dcbc82bb: Generated new TSK 0
I20260924 11:39:04.672236 26957 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:39:04.677695 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.1:0
--local_ip_for_outbound_sockets=127.6.212.1
--webserver_interface=127.6.212.1
--webserver_port=0
--tserver_master_addrs=127.6.212.62:35771
--builtin_ntp_servers=127.6.212.20:45907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 11:39:05.056137 26970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:39:05.056468 26970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:39:05.056634 26970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:39:05.063936 26970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:39:05.064085 26970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:39:05.064265 26970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.1
I20260924 11:39:05.073765 26970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:45907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.212.1
--webserver_port=0
--tserver_master_addrs=127.6.212.62:35771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.1
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:39:05.075192 26970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:39:05.077126 26970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:39:05.091048 26975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:05.090999 26976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:05.091002 26978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:39:05.091853 26970 server_base.cc:1034] running on GCE node
I20260924 11:39:05.092525 26970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:39:05.093568 26970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:39:05.094843 26970 hybrid_clock.cc:648] HybridClock initialized: now 1790249945094779 us; error 64 us; skew 500 ppm
I20260924 11:39:05.095216 26970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:39:05.097667 26970 webserver.cc:460] Webserver started at http://127.6.212.1:38691/ using document root <none> and password file <none>
I20260924 11:39:05.098446 26970 fs_manager.cc:359] Metadata directory not provided
I20260924 11:39:05.098562 26970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:39:05.098905 26970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:39:05.101946 26970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-0/data/instance:
uuid: "0993799c1b0343adac74fe5e5e52bec3"
format_stamp: "Formatted at 2026-09-24 11:39:05 on dist-test-slave-cq7c"
server_key: "c946da80546fd21cad7e718de109faf9"
server_key_iv: "10ec47674b46d46b7eda719bfc5dc98e"
server_key_version: "encryptionkey@0"
I20260924 11:39:05.102772 26970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance:
uuid: "0993799c1b0343adac74fe5e5e52bec3"
format_stamp: "Formatted at 2026-09-24 11:39:05 on dist-test-slave-cq7c"
server_key: "c946da80546fd21cad7e718de109faf9"
server_key_iv: "10ec47674b46d46b7eda719bfc5dc98e"
server_key_version: "encryptionkey@0"
I20260924 11:39:05.108248 26970 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260924 11:39:05.111963 26984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:39:05.113410 26970 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:39:05.113588 26970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-0/wal
uuid: "0993799c1b0343adac74fe5e5e52bec3"
format_stamp: "Formatted at 2026-09-24 11:39:05 on dist-test-slave-cq7c"
server_key: "c946da80546fd21cad7e718de109faf9"
server_key_iv: "10ec47674b46d46b7eda719bfc5dc98e"
server_key_version: "encryptionkey@0"
I20260924 11:39:05.113828 26970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:39:05.148427 26970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:39:05.149329 26970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:39:05.149617 26970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:39:05.150703 26970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:39:05.152359 26970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:39:05.152491 26970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:39:05.152631 26970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:39:05.152722 26970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:39:05.187593 26970 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.1:45647
I20260924 11:39:05.187678 27096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.1:45647 every 8 connection(s)
I20260924 11:39:05.188969 26970 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-0/data/info.pb
I20260924 11:39:05.194487 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 26970
I20260924 11:39:05.194655 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-0/wal/instance
I20260924 11:39:05.195108 6992 external_mini_cluster.cc:1428] Setting key e36cf0aa7e45f83687545ba7cb23d0d3
I20260924 11:39:05.197757 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.2:0
--local_ip_for_outbound_sockets=127.6.212.2
--webserver_interface=127.6.212.2
--webserver_port=0
--tserver_master_addrs=127.6.212.62:35771
--builtin_ntp_servers=127.6.212.20:45907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 11:39:05.202152 27099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:05.191770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:44085 (local address 127.6.212.62:35771)
0924 11:39:05.192238 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:05.192243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:05.193579 (+ 1336us) server_negotiation.cc:407] Connection header received
0924 11:39:05.195606 (+ 2027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:05.195611 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:05.195702 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:05.195835 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:05.196924 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:05.197648 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:05.199261 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:05.199778 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:05.199925 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:05.201307 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:05.201313 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:05.201318 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:39:05.201347 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:05.201387 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:39:05.201400 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:05.201497 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:05.201724 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 11:39:05.201817 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":231,"threads_started":1}
I20260924 11:39:05.203271 27100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:05.192466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35771 (local address 127.6.212.1:44085)
0924 11:39:05.193418 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:05.193460 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 11:39:05.195275 (+ 1815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:39:05.196078 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:39:05.196091 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:39:05.196182 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:39:05.196749 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:05.196765 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:05.197810 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:05.197815 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:05.199110 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:05.199126 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:05.200009 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:05.200016 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:05.200244 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:05.200983 (+ 739us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:39:05.200986 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:39:05.201026 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:39:05.201099 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:39:05.201510 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:39:05.201523 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:39:05.201531 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 11:39:05.201714 (+ 183us) client_negotiation.cc:240] Negotiation successful
0924 11:39:05.202111 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":335,"threads_started":1}
I20260924 11:39:05.204847 27097 heartbeater.cc:347] Connected to a master server at 127.6.212.62:35771
I20260924 11:39:05.205157 27097 heartbeater.cc:464] Registering TS with master...
I20260924 11:39:05.205901 27097 heartbeater.cc:511] Master 127.6.212.62:35771 requested a full tablet report, sending...
I20260924 11:39:05.208122 26911 ts_manager.cc:194] Registered new tserver with Master: 0993799c1b0343adac74fe5e5e52bec3 (127.6.212.1:45647)
I20260924 11:39:05.210253 26911 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.212.1:44085
W20260924 11:39:05.530367 27101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:39:05.530654 27101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:39:05.530741 27101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:39:05.538326 27101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:39:05.538445 27101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:39:05.538555 27101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.2
I20260924 11:39:05.547937 27101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:45907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.212.2
--webserver_port=0
--tserver_master_addrs=127.6.212.62:35771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.2
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:39:05.549314 27101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:39:05.551096 27101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:39:05.564847 27109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:05.564862 27107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:05.565356 27106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:39:05.565658 27101 server_base.cc:1034] running on GCE node
I20260924 11:39:05.566242 27101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:39:05.567188 27101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:39:05.568421 27101 hybrid_clock.cc:648] HybridClock initialized: now 1790249945568340 us; error 105 us; skew 500 ppm
I20260924 11:39:05.568739 27101 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:39:05.570899 27101 webserver.cc:460] Webserver started at http://127.6.212.2:32973/ using document root <none> and password file <none>
I20260924 11:39:05.571652 27101 fs_manager.cc:359] Metadata directory not provided
I20260924 11:39:05.571786 27101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:39:05.572108 27101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:39:05.575006 27101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-1/data/instance:
uuid: "4bfa38ed7208453cb9bf7785e446e0a3"
format_stamp: "Formatted at 2026-09-24 11:39:05 on dist-test-slave-cq7c"
server_key: "c0df5b71fb6d9d02ae57f86ea49323ad"
server_key_iv: "7bd51d303737f993a9e75c366956e40d"
server_key_version: "encryptionkey@0"
I20260924 11:39:05.575811 27101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance:
uuid: "4bfa38ed7208453cb9bf7785e446e0a3"
format_stamp: "Formatted at 2026-09-24 11:39:05 on dist-test-slave-cq7c"
server_key: "c0df5b71fb6d9d02ae57f86ea49323ad"
server_key_iv: "7bd51d303737f993a9e75c366956e40d"
server_key_version: "encryptionkey@0"
I20260924 11:39:05.580953 27101 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 11:39:05.585057 27115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:39:05.586634 27101 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:39:05.586789 27101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-1/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-1/wal
uuid: "4bfa38ed7208453cb9bf7785e446e0a3"
format_stamp: "Formatted at 2026-09-24 11:39:05 on dist-test-slave-cq7c"
server_key: "c0df5b71fb6d9d02ae57f86ea49323ad"
server_key_iv: "7bd51d303737f993a9e75c366956e40d"
server_key_version: "encryptionkey@0"
I20260924 11:39:05.587033 27101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:39:05.603027 27101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:39:05.604015 27101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:39:05.604293 27101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:39:05.605398 27101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:39:05.606801 27101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:39:05.606940 27101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:39:05.607074 27101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:39:05.607163 27101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:39:05.641614 27101 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.2:37335
I20260924 11:39:05.641782 27227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.2:37335 every 8 connection(s)
I20260924 11:39:05.642972 27101 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-1/data/info.pb
I20260924 11:39:05.646430 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 27101
I20260924 11:39:05.646620 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-1/wal/instance
I20260924 11:39:05.647212 6992 external_mini_cluster.cc:1428] Setting key eaf5715bd147b728847dd2448eb90987
I20260924 11:39:05.650444 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.212.3:0
--local_ip_for_outbound_sockets=127.6.212.3
--webserver_interface=127.6.212.3
--webserver_port=0
--tserver_master_addrs=127.6.212.62:35771
--builtin_ntp_servers=127.6.212.20:45907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 11:39:05.657289 27099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:05.645559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.2:40901 (local address 127.6.212.62:35771)
0924 11:39:05.645766 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:05.645770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:05.647383 (+ 1613us) server_negotiation.cc:407] Connection header received
0924 11:39:05.649667 (+ 2284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:05.649671 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:05.649735 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:05.649808 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:05.651233 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:05.651990 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:05.654423 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:05.654812 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:05.654905 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:05.656580 (+ 1675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:05.656585 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:05.656588 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:39:05.656615 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:05.656647 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:39:05.656659 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:05.656728 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:05.656978 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 11:39:05.657060 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 11:39:05.658835 27230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:05.646088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35771 (local address 127.6.212.2:40901)
0924 11:39:05.647183 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:05.647236 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 11:39:05.649367 (+ 2131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:39:05.650087 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:39:05.650102 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:39:05.650226 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:39:05.651049 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:05.651076 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:05.652143 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:05.652152 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:05.654269 (+ 2117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:05.654285 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:05.655026 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:05.655036 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:05.655398 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:05.656218 (+ 820us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:39:05.656224 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:39:05.656285 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:39:05.656388 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:39:05.656755 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:39:05.656767 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:39:05.656778 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 11:39:05.656967 (+ 189us) client_negotiation.cc:240] Negotiation successful
0924 11:39:05.657419 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":350,"threads_started":1}
I20260924 11:39:05.660300 27228 heartbeater.cc:347] Connected to a master server at 127.6.212.62:35771
I20260924 11:39:05.660626 27228 heartbeater.cc:464] Registering TS with master...
I20260924 11:39:05.661337 27228 heartbeater.cc:511] Master 127.6.212.62:35771 requested a full tablet report, sending...
I20260924 11:39:05.662951 26911 ts_manager.cc:194] Registered new tserver with Master: 4bfa38ed7208453cb9bf7785e446e0a3 (127.6.212.2:37335)
I20260924 11:39:05.663868 26911 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.212.2:40901
W20260924 11:39:06.004703 27231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:39:06.004992 27231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:39:06.005154 27231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:39:06.012529 27231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:39:06.012645 27231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:39:06.012832 27231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.212.3
I20260924 11:39:06.022681 27231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:45907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.212.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.212.3
--webserver_port=0
--tserver_master_addrs=127.6.212.62:35771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.212.3
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:39:06.024235 27231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:39:06.026119 27231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:39:06.039916 27236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:06.039892 27237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:06.039856 27239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:39:06.040668 27231 server_base.cc:1034] running on GCE node
I20260924 11:39:06.041296 27231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:39:06.042297 27231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:39:06.043519 27231 hybrid_clock.cc:648] HybridClock initialized: now 1790249946043470 us; error 43 us; skew 500 ppm
I20260924 11:39:06.043968 27231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 11:39:06.046375 27231 webserver.cc:460] Webserver started at http://127.6.212.3:41597/ using document root <none> and password file <none>
I20260924 11:39:06.047093 27231 fs_manager.cc:359] Metadata directory not provided
I20260924 11:39:06.047228 27231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:39:06.047899 27231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:39:06.051095 27231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-2/data/instance:
uuid: "484a1904825c40c882b646a103a92741"
format_stamp: "Formatted at 2026-09-24 11:39:06 on dist-test-slave-cq7c"
server_key: "018e1f48262a793467458a5060d1d4b6"
server_key_iv: "15eaecd87af598b9d66b10013c8856a3"
server_key_version: "encryptionkey@0"
I20260924 11:39:06.052070 27231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance:
uuid: "484a1904825c40c882b646a103a92741"
format_stamp: "Formatted at 2026-09-24 11:39:06 on dist-test-slave-cq7c"
server_key: "018e1f48262a793467458a5060d1d4b6"
server_key_iv: "15eaecd87af598b9d66b10013c8856a3"
server_key_version: "encryptionkey@0"
I20260924 11:39:06.058418 27231 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 11:39:06.062605 27245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:39:06.064105 27231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:39:06.064288 27231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-2/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-2/wal
uuid: "484a1904825c40c882b646a103a92741"
format_stamp: "Formatted at 2026-09-24 11:39:06 on dist-test-slave-cq7c"
server_key: "018e1f48262a793467458a5060d1d4b6"
server_key_iv: "15eaecd87af598b9d66b10013c8856a3"
server_key_version: "encryptionkey@0"
I20260924 11:39:06.064523 27231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:39:06.085018 27231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:39:06.086012 27231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:39:06.086333 27231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:39:06.087529 27231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 11:39:06.089234 27231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 11:39:06.089376 27231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 11:39:06.089525 27231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 11:39:06.089625 27231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 11:39:06.127933 27231 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.3:43963
I20260924 11:39:06.127974 27357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.3:43963 every 8 connection(s)
I20260924 11:39:06.129624 27231 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-2/data/info.pb
I20260924 11:39:06.134028 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 27231
I20260924 11:39:06.134229 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-2/wal/instance
I20260924 11:39:06.134832 6992 external_mini_cluster.cc:1428] Setting key 2ba435620c00531e4d6fa07a4afbfe9c
I20260924 11:39:06.143852 27099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:06.132410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:45059 (local address 127.6.212.62:35771)
0924 11:39:06.132581 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:06.132585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:06.134272 (+ 1687us) server_negotiation.cc:407] Connection header received
0924 11:39:06.136887 (+ 2615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:06.136894 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:06.136992 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:06.137102 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:06.138379 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:06.139177 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:06.141055 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:06.141467 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:06.141574 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:06.143075 (+ 1501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:06.143080 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:06.143084 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:39:06.143102 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:06.143133 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:39:06.143144 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:06.143225 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:06.143520 (+ 295us) server_negotiation.cc:299] Negotiation successful
0924 11:39:06.143606 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 11:39:06.145506 27360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:06.132958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:35771 (local address 127.6.212.3:45059)
0924 11:39:06.134075 (+ 1117us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:06.134133 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 11:39:06.136529 (+ 2396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:39:06.137370 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:39:06.137379 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:39:06.137470 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:39:06.138222 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:06.138240 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:06.139315 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:06.139319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:06.140899 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:06.140911 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:06.141677 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:06.141683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:06.141980 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:06.142757 (+ 777us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 11:39:06.142760 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:39:06.142805 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 11:39:06.142888 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:39:06.143250 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:39:06.143258 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:39:06.143264 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 11:39:06.143502 (+ 238us) client_negotiation.cc:240] Negotiation successful
0924 11:39:06.144157 (+ 655us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":311,"threads_started":1}
I20260924 11:39:06.146991 27358 heartbeater.cc:347] Connected to a master server at 127.6.212.62:35771
I20260924 11:39:06.147315 27358 heartbeater.cc:464] Registering TS with master...
I20260924 11:39:06.148221 27358 heartbeater.cc:511] Master 127.6.212.62:35771 requested a full tablet report, sending...
I20260924 11:39:06.149818 26911 ts_manager.cc:194] Registered new tserver with Master: 484a1904825c40c882b646a103a92741 (127.6.212.3:43963)
I20260924 11:39:06.150590 6992 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 11:39:06.150753 26911 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.212.3:45059
I20260924 11:39:06.162072 27099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:06.155169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33206 (local address 127.6.212.62:35771)
0924 11:39:06.155408 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:06.155418 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:06.155992 (+ 574us) server_negotiation.cc:407] Connection header received
0924 11:39:06.156222 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:06.156317 (+ 95us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:06.156414 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:06.156562 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:06.157081 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:06.157744 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:06.159148 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:06.159950 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:06.160121 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:06.161356 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:06.161361 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:06.161364 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:39:06.161381 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:06.161410 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:39:06.161423 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:06.161526 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:06.161672 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 11:39:06.161772 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 11:39:06.166028 26911 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33206:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 11:39:06.172931 26911 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 11:39:06.206911 27374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:06.191276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47978 (local address 127.6.212.3:43963)
0924 11:39:06.195130 (+ 3854us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:06.195144 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:06.195213 (+ 69us) server_negotiation.cc:407] Connection header received
0924 11:39:06.195377 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:06.195389 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:06.195778 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:06.196058 (+ 280us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:39:06.197444 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:06.199848 (+ 2404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:06.203302 (+ 3454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:06.204940 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:06.205091 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:06.205412 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:06.206279 (+ 867us) server_negotiation.cc:299] Negotiation successful
0924 11:39:06.206481 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3636,"thread_start_us":3475,"threads_started":1}
I20260924 11:39:06.207036 27373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:06.186776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.3:43963 (local address 127.0.0.1:47978)
0924 11:39:06.188955 (+ 2179us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:06.188980 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 11:39:06.189092 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:39:06.196520 (+ 7428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:39:06.196534 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:39:06.196584 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:39:06.197097 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:06.197127 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:06.200095 (+ 2968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:06.200105 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:06.203075 (+ 2970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:06.203089 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:06.205558 (+ 2469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:06.205566 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:06.205928 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:06.206047 (+ 119us) client_negotiation.cc:769] Sending connection context
0924 11:39:06.206208 (+ 161us) client_negotiation.cc:240] Negotiation successful
0924 11:39:06.206559 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2003,"mutex_wait_us":197,"thread_start_us":506,"threads_started":1}
I20260924 11:39:06.211148 27370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:06.188313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:45647 (local address 127.0.0.1:59476)
0924 11:39:06.200419 (+ 12106us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:06.200457 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 11:39:06.200624 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:39:06.203406 (+ 2782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:39:06.203413 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:39:06.203443 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:39:06.203802 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:06.203815 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:06.205801 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:06.205806 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:06.208254 (+ 2448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:06.208270 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:06.210245 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:06.210252 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:06.210460 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:06.210508 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 11:39:06.210615 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 11:39:06.210748 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11880,"mutex_wait_us":38,"thread_start_us":296,"threads_started":1}
I20260924 11:39:06.211392 27371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:06.186788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.2:37335 (local address 127.0.0.1:48462)
0924 11:39:06.187766 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:06.187835 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0924 11:39:06.188044 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:39:06.202362 (+ 14318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:39:06.202370 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:39:06.202405 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:39:06.202868 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:06.202881 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:06.205179 (+ 2298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:06.205185 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:06.207658 (+ 2473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:06.207672 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:06.210459 (+ 2787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:06.210466 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:06.210707 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:06.210742 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 11:39:06.210814 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 11:39:06.210948 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"mutex_wait_us":32,"thread_start_us":13953,"threads_started":1}
I20260924 11:39:06.211465 27375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:06.191890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59476 (local address 127.6.212.1:45647)
0924 11:39:06.202661 (+ 10771us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:06.202673 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:06.202707 (+ 34us) server_negotiation.cc:407] Connection header received
0924 11:39:06.202824 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:06.202833 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:06.203084 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:06.203452 (+ 368us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:39:06.203992 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:06.205643 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:06.208503 (+ 2860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:06.210015 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:06.210167 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:06.210475 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:06.210697 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 11:39:06.210899 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10570,"thread_start_us":10440,"threads_started":1}
I20260924 11:39:06.211673 27293 tablet_service.cc:1467] Processing CreateTablet for tablet 67dbaa28babb4f64810d45c87198df0d (DEFAULT_TABLE table=test-table [id=8264f64386f345e09418003c14f4822d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:39:06.213366 27097 heartbeater.cc:503] Master 127.6.212.62:35771 was elected leader, sending a full tablet report...
I20260924 11:39:06.213491 27293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67dbaa28babb4f64810d45c87198df0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:39:06.215524 27372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:06.187380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48462 (local address 127.6.212.2:37335)
0924 11:39:06.201661 (+ 14281us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:06.201674 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:06.201708 (+ 34us) server_negotiation.cc:407] Connection header received
0924 11:39:06.201812 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:06.201819 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:06.202092 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:06.203098 (+ 1006us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:39:06.203386 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:06.205014 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:06.207876 (+ 2862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:06.210212 (+ 2336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:06.211772 (+ 1560us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:06.212037 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:06.212092 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 11:39:06.212217 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14094,"thread_start_us":14606,"threads_started":1}
I20260924 11:39:06.215694 27032 tablet_service.cc:1467] Processing CreateTablet for tablet 67dbaa28babb4f64810d45c87198df0d (DEFAULT_TABLE table=test-table [id=8264f64386f345e09418003c14f4822d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:39:06.217485 27032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67dbaa28babb4f64810d45c87198df0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:39:06.219671 27163 tablet_service.cc:1467] Processing CreateTablet for tablet 67dbaa28babb4f64810d45c87198df0d (DEFAULT_TABLE table=test-table [id=8264f64386f345e09418003c14f4822d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 11:39:06.221254 27163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67dbaa28babb4f64810d45c87198df0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:39:06.229759 27376 tablet_bootstrap.cc:492] T 67dbaa28babb4f64810d45c87198df0d P 484a1904825c40c882b646a103a92741: Bootstrap starting.
I20260924 11:39:06.231953 27377 tablet_bootstrap.cc:492] T 67dbaa28babb4f64810d45c87198df0d P 0993799c1b0343adac74fe5e5e52bec3: Bootstrap starting.
I20260924 11:39:06.235688 27376 tablet_bootstrap.cc:654] T 67dbaa28babb4f64810d45c87198df0d P 484a1904825c40c882b646a103a92741: Neither blocks nor log segments found. Creating new log.
I20260924 11:39:06.236042 27377 tablet_bootstrap.cc:654] T 67dbaa28babb4f64810d45c87198df0d P 0993799c1b0343adac74fe5e5e52bec3: Neither blocks nor log segments found. Creating new log.
I20260924 11:39:06.237123 27378 tablet_bootstrap.cc:492] T 67dbaa28babb4f64810d45c87198df0d P 4bfa38ed7208453cb9bf7785e446e0a3: Bootstrap starting.
I20260924 11:39:06.237114 27376 log.cc:824] T 67dbaa28babb4f64810d45c87198df0d P 484a1904825c40c882b646a103a92741: Log is configured to *not* fsync() on all Append() calls
I20260924 11:39:06.237407 27377 log.cc:824] T 67dbaa28babb4f64810d45c87198df0d P 0993799c1b0343adac74fe5e5e52bec3: Log is configured to *not* fsync() on all Append() calls
I20260924 11:39:06.241322 27377 tablet_bootstrap.cc:492] T 67dbaa28babb4f64810d45c87198df0d P 0993799c1b0343adac74fe5e5e52bec3: No bootstrap required, opened a new log
I20260924 11:39:06.241772 27377 ts_tablet_manager.cc:1397] T 67dbaa28babb4f64810d45c87198df0d P 0993799c1b0343adac74fe5e5e52bec3: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20260924 11:39:06.242266 27378 tablet_bootstrap.cc:654] T 67dbaa28babb4f64810d45c87198df0d P 4bfa38ed7208453cb9bf7785e446e0a3: Neither blocks nor log segments found. Creating new log.
I20260924 11:39:06.242333 27376 tablet_bootstrap.cc:492] T 67dbaa28babb4f64810d45c87198df0d P 484a1904825c40c882b646a103a92741: No bootstrap required, opened a new log
I20260924 11:39:06.242619 27376 ts_tablet_manager.cc:1397] T 67dbaa28babb4f64810d45c87198df0d P 484a1904825c40c882b646a103a92741: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.003s
I20260924 11:39:06.243754 27378 log.cc:824] T 67dbaa28babb4f64810d45c87198df0d P 4bfa38ed7208453cb9bf7785e446e0a3: Log is configured to *not* fsync() on all Append() calls
I20260924 11:39:06.247615 27378 tablet_bootstrap.cc:492] T 67dbaa28babb4f64810d45c87198df0d P 4bfa38ed7208453cb9bf7785e446e0a3: No bootstrap required, opened a new log
I20260924 11:39:06.247924 27378 ts_tablet_manager.cc:1397] T 67dbaa28babb4f64810d45c87198df0d P 4bfa38ed7208453cb9bf7785e446e0a3: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20260924 11:39:06.250530 27377 raft_consensus.cc:348] T 67dbaa28babb4f64810d45c87198df0d P 0993799c1b0343adac74fe5e5e52bec3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484a1904825c40c882b646a103a92741" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43963 } } peers { permanent_uuid: "4bfa38ed7208453cb9bf7785e446e0a3" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37335 } } peers { permanent_uuid: "0993799c1b0343adac74fe5e5e52bec3" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 45647 } }
I20260924 11:39:06.251003 27377 raft_consensus.cc:374] T 67dbaa28babb4f64810d45c87198df0d P 0993799c1b0343adac74fe5e5e52bec3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:39:06.251138 27377 raft_consensus.cc:729] T 67dbaa28babb4f64810d45c87198df0d P 0993799c1b0343adac74fe5e5e52bec3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0993799c1b0343adac74fe5e5e52bec3, State: Initialized, Role: FOLLOWER
I20260924 11:39:06.251935 27377 consensus_queue.cc:260] T 67dbaa28babb4f64810d45c87198df0d P 0993799c1b0343adac74fe5e5e52bec3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484a1904825c40c882b646a103a92741" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43963 } } peers { permanent_uuid: "4bfa38ed7208453cb9bf7785e446e0a3" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37335 } } peers { permanent_uuid: "0993799c1b0343adac74fe5e5e52bec3" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 45647 } }
I20260924 11:39:06.254447 27376 raft_consensus.cc:348] T 67dbaa28babb4f64810d45c87198df0d P 484a1904825c40c882b646a103a92741 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484a1904825c40c882b646a103a92741" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43963 } } peers { permanent_uuid: "4bfa38ed7208453cb9bf7785e446e0a3" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37335 } } peers { permanent_uuid: "0993799c1b0343adac74fe5e5e52bec3" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 45647 } }
I20260924 11:39:06.254941 27376 raft_consensus.cc:374] T 67dbaa28babb4f64810d45c87198df0d P 484a1904825c40c882b646a103a92741 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:39:06.255118 27376 raft_consensus.cc:729] T 67dbaa28babb4f64810d45c87198df0d P 484a1904825c40c882b646a103a92741 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 484a1904825c40c882b646a103a92741, State: Initialized, Role: FOLLOWER
I20260924 11:39:06.255932 27376 consensus_queue.cc:260] T 67dbaa28babb4f64810d45c87198df0d P 484a1904825c40c882b646a103a92741 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484a1904825c40c882b646a103a92741" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43963 } } peers { permanent_uuid: "4bfa38ed7208453cb9bf7785e446e0a3" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37335 } } peers { permanent_uuid: "0993799c1b0343adac74fe5e5e52bec3" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 45647 } }
I20260924 11:39:06.256608 27377 ts_tablet_manager.cc:1428] T 67dbaa28babb4f64810d45c87198df0d P 0993799c1b0343adac74fe5e5e52bec3: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260924 11:39:06.257736 27358 heartbeater.cc:503] Master 127.6.212.62:35771 was elected leader, sending a full tablet report...
I20260924 11:39:06.258450 27376 ts_tablet_manager.cc:1428] T 67dbaa28babb4f64810d45c87198df0d P 484a1904825c40c882b646a103a92741: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260924 11:39:06.259965 27378 raft_consensus.cc:348] T 67dbaa28babb4f64810d45c87198df0d P 4bfa38ed7208453cb9bf7785e446e0a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484a1904825c40c882b646a103a92741" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43963 } } peers { permanent_uuid: "4bfa38ed7208453cb9bf7785e446e0a3" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37335 } } peers { permanent_uuid: "0993799c1b0343adac74fe5e5e52bec3" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 45647 } }
I20260924 11:39:06.260514 27378 raft_consensus.cc:374] T 67dbaa28babb4f64810d45c87198df0d P 4bfa38ed7208453cb9bf7785e446e0a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:39:06.260623 27378 raft_consensus.cc:729] T 67dbaa28babb4f64810d45c87198df0d P 4bfa38ed7208453cb9bf7785e446e0a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bfa38ed7208453cb9bf7785e446e0a3, State: Initialized, Role: FOLLOWER
I20260924 11:39:06.261301 27378 consensus_queue.cc:260] T 67dbaa28babb4f64810d45c87198df0d P 4bfa38ed7208453cb9bf7785e446e0a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484a1904825c40c882b646a103a92741" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43963 } } peers { permanent_uuid: "4bfa38ed7208453cb9bf7785e446e0a3" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37335 } } peers { permanent_uuid: "0993799c1b0343adac74fe5e5e52bec3" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 45647 } }
I20260924 11:39:06.262574 27228 heartbeater.cc:503] Master 127.6.212.62:35771 was elected leader, sending a full tablet report...
I20260924 11:39:06.263239 27378 ts_tablet_manager.cc:1428] T 67dbaa28babb4f64810d45c87198df0d P 4bfa38ed7208453cb9bf7785e446e0a3: Time spent starting tablet: real 0.015s user 0.010s sys 0.005s
W20260924 11:39:06.381515 27359 tablet.cc:2369] T 67dbaa28babb4f64810d45c87198df0d P 484a1904825c40c882b646a103a92741: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:39:06.395087 27229 tablet.cc:2369] T 67dbaa28babb4f64810d45c87198df0d P 4bfa38ed7208453cb9bf7785e446e0a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 11:39:06.441627 27098 tablet.cc:2369] T 67dbaa28babb4f64810d45c87198df0d P 0993799c1b0343adac74fe5e5e52bec3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:39:06.452729 27382 raft_consensus.cc:482] T 67dbaa28babb4f64810d45c87198df0d P 0993799c1b0343adac74fe5e5e52bec3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:39:06.453071 27382 raft_consensus.cc:504] T 67dbaa28babb4f64810d45c87198df0d P 0993799c1b0343adac74fe5e5e52bec3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484a1904825c40c882b646a103a92741" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43963 } } peers { permanent_uuid: "4bfa38ed7208453cb9bf7785e446e0a3" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37335 } } peers { permanent_uuid: "0993799c1b0343adac74fe5e5e52bec3" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 45647 } }
I20260924 11:39:06.454929 27382 leader_election.cc:288] T 67dbaa28babb4f64810d45c87198df0d P 0993799c1b0343adac74fe5e5e52bec3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 484a1904825c40c882b646a103a92741 (127.6.212.3:43963), 4bfa38ed7208453cb9bf7785e446e0a3 (127.6.212.2:37335)
I20260924 11:39:06.457628 27383 raft_consensus.cc:482] T 67dbaa28babb4f64810d45c87198df0d P 484a1904825c40c882b646a103a92741 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 11:39:06.457965 27383 raft_consensus.cc:504] T 67dbaa28babb4f64810d45c87198df0d P 484a1904825c40c882b646a103a92741 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484a1904825c40c882b646a103a92741" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43963 } } peers { permanent_uuid: "4bfa38ed7208453cb9bf7785e446e0a3" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37335 } } peers { permanent_uuid: "0993799c1b0343adac74fe5e5e52bec3" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 45647 } }
I20260924 11:39:06.460116 27383 leader_election.cc:288] T 67dbaa28babb4f64810d45c87198df0d P 484a1904825c40c882b646a103a92741 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bfa38ed7208453cb9bf7785e446e0a3 (127.6.212.2:37335), 0993799c1b0343adac74fe5e5e52bec3 (127.6.212.1:45647)
I20260924 11:39:06.464004 27372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:06.455700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:48269 (local address 127.6.212.2:37335)
0924 11:39:06.455988 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:06.455995 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:06.456350 (+ 355us) server_negotiation.cc:407] Connection header received
0924 11:39:06.456594 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:06.456600 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:06.456694 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:06.456790 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:39:06.457467 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:06.458490 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:06.461397 (+ 2907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:06.462687 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:06.462810 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:06.462904 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:06.463549 (+ 645us) server_negotiation.cc:299] Negotiation successful
0924 11:39:06.463716 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260924 11:39:06.464043 27374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:06.455302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.1:50279 (local address 127.6.212.3:43963)
0924 11:39:06.455516 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:06.455522 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:06.456803 (+ 1281us) server_negotiation.cc:407] Connection header received
0924 11:39:06.456900 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:06.456907 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:06.456990 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:06.457092 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:39:06.457784 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:06.458837 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:06.461550 (+ 2713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:06.462895 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:06.463014 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:06.463108 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:06.463708 (+ 600us) server_negotiation.cc:299] Negotiation successful
0924 11:39:06.463823 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":91}
I20260924 11:39:06.464761 27386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:06.455332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.3:43963 (local address 127.6.212.1:50279)
0924 11:39:06.456622 (+ 1290us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:06.456640 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:39:06.456790 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:39:06.457113 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:39:06.457116 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:39:06.457138 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:39:06.457568 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:06.457582 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:06.458980 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:06.458986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:06.461372 (+ 2386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:06.461384 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:06.463248 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:06.463258 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:06.463515 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:06.463573 (+ 58us) client_negotiation.cc:769] Sending connection context
0924 11:39:06.464255 (+ 682us) client_negotiation.cc:240] Negotiation successful
0924 11:39:06.464384 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1160,"mutex_wait_us":610,"thread_start_us":663,"threads_started":1}
I20260924 11:39:06.465209 27385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:06.455325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.2:37335 (local address 127.6.212.1:48269)
0924 11:39:06.456190 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:06.456220 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 11:39:06.456418 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:39:06.456835 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:39:06.456842 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:39:06.456874 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:39:06.457311 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:06.457322 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:06.458628 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:06.458633 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:06.461184 (+ 2551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:06.461200 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:06.463063 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:06.463069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:06.463331 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:06.463386 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 11:39:06.464489 (+ 1103us) client_negotiation.cc:240] Negotiation successful
0924 11:39:06.464606 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":151,"thread_start_us":351,"threads_started":1}
I20260924 11:39:06.465989 27313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dbaa28babb4f64810d45c87198df0d" candidate_uuid: "0993799c1b0343adac74fe5e5e52bec3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "484a1904825c40c882b646a103a92741" is_pre_election: true
I20260924 11:39:06.466200 27183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dbaa28babb4f64810d45c87198df0d" candidate_uuid: "0993799c1b0343adac74fe5e5e52bec3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bfa38ed7208453cb9bf7785e446e0a3" is_pre_election: true
I20260924 11:39:06.466500 27313 raft_consensus.cc:2454] T 67dbaa28babb4f64810d45c87198df0d P 484a1904825c40c882b646a103a92741 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0993799c1b0343adac74fe5e5e52bec3 in term 0.
I20260924 11:39:06.466713 27183 raft_consensus.cc:2454] T 67dbaa28babb4f64810d45c87198df0d P 4bfa38ed7208453cb9bf7785e446e0a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0993799c1b0343adac74fe5e5e52bec3 in term 0.
I20260924 11:39:06.467329 26988 leader_election.cc:302] T 67dbaa28babb4f64810d45c87198df0d P 0993799c1b0343adac74fe5e5e52bec3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0993799c1b0343adac74fe5e5e52bec3, 484a1904825c40c882b646a103a92741; no voters:
I20260924 11:39:06.468092 27382 raft_consensus.cc:2781] T 67dbaa28babb4f64810d45c87198df0d P 0993799c1b0343adac74fe5e5e52bec3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:39:06.468305 27382 raft_consensus.cc:482] T 67dbaa28babb4f64810d45c87198df0d P 0993799c1b0343adac74fe5e5e52bec3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:39:06.468441 27382 raft_consensus.cc:3037] T 67dbaa28babb4f64810d45c87198df0d P 0993799c1b0343adac74fe5e5e52bec3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:39:06.469612 27360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:06.460349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.2:37335 (local address 127.6.212.3:50331)
0924 11:39:06.460741 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:06.460762 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 11:39:06.460874 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:39:06.462527 (+ 1653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:39:06.462533 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:39:06.462562 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:39:06.462894 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:06.462906 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:06.464695 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:06.464700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:06.467142 (+ 2442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:06.467156 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:06.468823 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:06.468832 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:06.469021 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:06.469052 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 11:39:06.469146 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 11:39:06.469299 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":171}
I20260924 11:39:06.469995 27375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:06.461548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:58449 (local address 127.6.212.1:45647)
0924 11:39:06.461762 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:06.461769 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:06.461788 (+ 19us) server_negotiation.cc:407] Connection header received
0924 11:39:06.461853 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:06.461857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:06.461947 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:06.462035 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:39:06.463076 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:06.464455 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:06.467848 (+ 3393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:06.469131 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:06.469274 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:06.469372 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:06.469633 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 11:39:06.469763 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":66}
I20260924 11:39:06.470044 27387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:06.460686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:45647 (local address 127.6.212.3:58449)
0924 11:39:06.461442 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:06.461459 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 11:39:06.461562 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:39:06.462265 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:39:06.462270 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:39:06.462293 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 11:39:06.462869 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:06.462881 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:06.464632 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:06.464640 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:06.467509 (+ 2869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:06.467523 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:06.469311 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:06.469317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:06.469482 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:06.469517 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 11:39:06.469604 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 11:39:06.469720 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"mutex_wait_us":438,"thread_start_us":413,"threads_started":1}
I20260924 11:39:06.470777 27388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:06.461182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.212.3:50331 (local address 127.6.212.2:37335)
0924 11:39:06.462183 (+ 1001us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:06.462188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:06.462213 (+ 25us) server_negotiation.cc:407] Connection header received
0924 11:39:06.462263 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:06.462268 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:06.462348 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:06.463038 (+ 690us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 11:39:06.463306 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:06.464562 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:06.467327 (+ 2765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:06.468618 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:06.468744 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:06.468867 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:06.470299 (+ 1432us) server_negotiation.cc:299] Negotiation successful
0924 11:39:06.470434 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":845,"thread_start_us":897,"threads_started":1}
I20260924 11:39:06.470923 27052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dbaa28babb4f64810d45c87198df0d" candidate_uuid: "484a1904825c40c882b646a103a92741" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0993799c1b0343adac74fe5e5e52bec3" is_pre_election: true
I20260924 11:39:06.471315 27183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dbaa28babb4f64810d45c87198df0d" candidate_uuid: "484a1904825c40c882b646a103a92741" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bfa38ed7208453cb9bf7785e446e0a3" is_pre_election: true
I20260924 11:39:06.471520 27183 raft_consensus.cc:2454] T 67dbaa28babb4f64810d45c87198df0d P 4bfa38ed7208453cb9bf7785e446e0a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 484a1904825c40c882b646a103a92741 in term 0.
I20260924 11:39:06.471912 27248 leader_election.cc:302] T 67dbaa28babb4f64810d45c87198df0d P 484a1904825c40c882b646a103a92741 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 484a1904825c40c882b646a103a92741, 4bfa38ed7208453cb9bf7785e446e0a3; no voters:
I20260924 11:39:06.471918 27382 raft_consensus.cc:504] T 67dbaa28babb4f64810d45c87198df0d P 0993799c1b0343adac74fe5e5e52bec3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484a1904825c40c882b646a103a92741" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43963 } } peers { permanent_uuid: "4bfa38ed7208453cb9bf7785e446e0a3" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37335 } } peers { permanent_uuid: "0993799c1b0343adac74fe5e5e52bec3" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 45647 } }
I20260924 11:39:06.472285 27383 raft_consensus.cc:2781] T 67dbaa28babb4f64810d45c87198df0d P 484a1904825c40c882b646a103a92741 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 11:39:06.472361 27052 raft_consensus.cc:2379] T 67dbaa28babb4f64810d45c87198df0d P 0993799c1b0343adac74fe5e5e52bec3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 484a1904825c40c882b646a103a92741 in current term 1: Already voted for candidate 0993799c1b0343adac74fe5e5e52bec3 in this term.
I20260924 11:39:06.472438 27383 raft_consensus.cc:482] T 67dbaa28babb4f64810d45c87198df0d P 484a1904825c40c882b646a103a92741 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 11:39:06.472543 27383 raft_consensus.cc:3037] T 67dbaa28babb4f64810d45c87198df0d P 484a1904825c40c882b646a103a92741 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:39:06.472729 27382 leader_election.cc:288] T 67dbaa28babb4f64810d45c87198df0d P 0993799c1b0343adac74fe5e5e52bec3 [CANDIDATE]: Term 1 election: Requested vote from peers 484a1904825c40c882b646a103a92741 (127.6.212.3:43963), 4bfa38ed7208453cb9bf7785e446e0a3 (127.6.212.2:37335)
I20260924 11:39:06.473337 27313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dbaa28babb4f64810d45c87198df0d" candidate_uuid: "0993799c1b0343adac74fe5e5e52bec3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "484a1904825c40c882b646a103a92741"
I20260924 11:39:06.473611 27183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dbaa28babb4f64810d45c87198df0d" candidate_uuid: "0993799c1b0343adac74fe5e5e52bec3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bfa38ed7208453cb9bf7785e446e0a3"
I20260924 11:39:06.473812 27183 raft_consensus.cc:3037] T 67dbaa28babb4f64810d45c87198df0d P 4bfa38ed7208453cb9bf7785e446e0a3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:39:06.475813 27383 raft_consensus.cc:504] T 67dbaa28babb4f64810d45c87198df0d P 484a1904825c40c882b646a103a92741 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484a1904825c40c882b646a103a92741" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43963 } } peers { permanent_uuid: "4bfa38ed7208453cb9bf7785e446e0a3" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37335 } } peers { permanent_uuid: "0993799c1b0343adac74fe5e5e52bec3" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 45647 } }
I20260924 11:39:06.476281 27313 raft_consensus.cc:2379] T 67dbaa28babb4f64810d45c87198df0d P 484a1904825c40c882b646a103a92741 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0993799c1b0343adac74fe5e5e52bec3 in current term 1: Already voted for candidate 484a1904825c40c882b646a103a92741 in this term.
I20260924 11:39:06.476965 27383 leader_election.cc:288] T 67dbaa28babb4f64810d45c87198df0d P 484a1904825c40c882b646a103a92741 [CANDIDATE]: Term 1 election: Requested vote from peers 4bfa38ed7208453cb9bf7785e446e0a3 (127.6.212.2:37335), 0993799c1b0343adac74fe5e5e52bec3 (127.6.212.1:45647)
I20260924 11:39:06.477052 27183 raft_consensus.cc:2454] T 67dbaa28babb4f64810d45c87198df0d P 4bfa38ed7208453cb9bf7785e446e0a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0993799c1b0343adac74fe5e5e52bec3 in term 1.
I20260924 11:39:06.477094 27182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dbaa28babb4f64810d45c87198df0d" candidate_uuid: "484a1904825c40c882b646a103a92741" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bfa38ed7208453cb9bf7785e446e0a3"
I20260924 11:39:06.477509 27182 raft_consensus.cc:2379] T 67dbaa28babb4f64810d45c87198df0d P 4bfa38ed7208453cb9bf7785e446e0a3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 484a1904825c40c882b646a103a92741 in current term 1: Already voted for candidate 0993799c1b0343adac74fe5e5e52bec3 in this term.
I20260924 11:39:06.477540 26987 leader_election.cc:302] T 67dbaa28babb4f64810d45c87198df0d P 0993799c1b0343adac74fe5e5e52bec3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0993799c1b0343adac74fe5e5e52bec3, 4bfa38ed7208453cb9bf7785e446e0a3; no voters:
I20260924 11:39:06.477939 27382 raft_consensus.cc:2781] T 67dbaa28babb4f64810d45c87198df0d P 0993799c1b0343adac74fe5e5e52bec3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:39:06.478076 27052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dbaa28babb4f64810d45c87198df0d" candidate_uuid: "484a1904825c40c882b646a103a92741" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0993799c1b0343adac74fe5e5e52bec3"
I20260924 11:39:06.478463 27382 raft_consensus.cc:686] T 67dbaa28babb4f64810d45c87198df0d P 0993799c1b0343adac74fe5e5e52bec3 [term 1 LEADER]: Becoming Leader. State: Replica: 0993799c1b0343adac74fe5e5e52bec3, State: Running, Role: LEADER
I20260924 11:39:06.478935 27382 consensus_queue.cc:237] T 67dbaa28babb4f64810d45c87198df0d P 0993799c1b0343adac74fe5e5e52bec3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484a1904825c40c882b646a103a92741" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43963 } } peers { permanent_uuid: "4bfa38ed7208453cb9bf7785e446e0a3" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37335 } } peers { permanent_uuid: "0993799c1b0343adac74fe5e5e52bec3" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 45647 } }
I20260924 11:39:06.481696 27249 leader_election.cc:302] T 67dbaa28babb4f64810d45c87198df0d P 484a1904825c40c882b646a103a92741 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 484a1904825c40c882b646a103a92741; no voters: 0993799c1b0343adac74fe5e5e52bec3, 4bfa38ed7208453cb9bf7785e446e0a3
I20260924 11:39:06.482024 27383 raft_consensus.cc:2726] T 67dbaa28babb4f64810d45c87198df0d P 484a1904825c40c882b646a103a92741 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260924 11:39:06.483731 26910 catalog_manager.cc:5495] T 67dbaa28babb4f64810d45c87198df0d P 0993799c1b0343adac74fe5e5e52bec3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0993799c1b0343adac74fe5e5e52bec3 (127.6.212.1). New cstate: current_term: 1 leader_uuid: "0993799c1b0343adac74fe5e5e52bec3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484a1904825c40c882b646a103a92741" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bfa38ed7208453cb9bf7785e446e0a3" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0993799c1b0343adac74fe5e5e52bec3" member_type: VOTER last_known_addr { host: "127.6.212.1" port: 45647 } health_report { overall_health: HEALTHY } } }
I20260924 11:39:06.510844 27375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:06.504675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34630 (local address 127.6.212.1:45647)
0924 11:39:06.504880 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:06.504886 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:06.505147 (+ 261us) server_negotiation.cc:407] Connection header received
0924 11:39:06.505381 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:06.505385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:06.505499 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:06.505621 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:39:06.506011 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:06.506779 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:06.508335 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:06.508997 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:06.509105 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:06.509637 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:39:06.510156 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:39:06.510264 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:06.510443 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 11:39:06.510511 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 11:39:06.519269 27313 raft_consensus.cc:1261] T 67dbaa28babb4f64810d45c87198df0d P 484a1904825c40c882b646a103a92741 [term 1 FOLLOWER]: Refusing update from remote peer 0993799c1b0343adac74fe5e5e52bec3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:39:06.519325 27182 raft_consensus.cc:1261] T 67dbaa28babb4f64810d45c87198df0d P 4bfa38ed7208453cb9bf7785e446e0a3 [term 1 FOLLOWER]: Refusing update from remote peer 0993799c1b0343adac74fe5e5e52bec3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:39:06.520205 27389 consensus_queue.cc:1035] T 67dbaa28babb4f64810d45c87198df0d P 0993799c1b0343adac74fe5e5e52bec3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "484a1904825c40c882b646a103a92741" member_type: VOTER last_known_addr { host: "127.6.212.3" port: 43963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:39:06.520648 27382 consensus_queue.cc:1035] T 67dbaa28babb4f64810d45c87198df0d P 0993799c1b0343adac74fe5e5e52bec3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bfa38ed7208453cb9bf7785e446e0a3" member_type: VOTER last_known_addr { host: "127.6.212.2" port: 37335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:39:06.533010 27392 mvcc.cc:205] Tried to move back new op lower bound from 7332863780932907008 to 7332863780781719552. Current Snapshot: MvccSnapshot[applied={T|T < 7332863780932907008}]
I20260924 11:39:06.534161 27394 mvcc.cc:205] Tried to move back new op lower bound from 7332863780932907008 to 7332863780781719552. Current Snapshot: MvccSnapshot[applied={T|T < 7332863780932907008}]
I20260924 11:39:06.547657 26910 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:39:06.547966 26910 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 11:39:06.552088 26910 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3d8a9ac84eaa4d0c954c8a87dcbc82bb: Sending DeleteTablet for 3 replicas of tablet 67dbaa28babb4f64810d45c87198df0d
I20260924 11:39:06.553471 27032 tablet_service.cc:1514] Processing DeleteTablet for tablet 67dbaa28babb4f64810d45c87198df0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:39:06 UTC) from {username='slave'} at 127.0.0.1:59476
I20260924 11:39:06.553438 27293 tablet_service.cc:1514] Processing DeleteTablet for tablet 67dbaa28babb4f64810d45c87198df0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:39:06 UTC) from {username='slave'} at 127.0.0.1:47978
I20260924 11:39:06.553459 27163 tablet_service.cc:1514] Processing DeleteTablet for tablet 67dbaa28babb4f64810d45c87198df0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 11:39:06 UTC) from {username='slave'} at 127.0.0.1:48462
I20260924 11:39:06.559662 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 26970
I20260924 11:39:06.568804 27402 tablet_replica.cc:330] T 67dbaa28babb4f64810d45c87198df0d P 484a1904825c40c882b646a103a92741: stopping tablet replica
I20260924 11:39:06.569523 27402 raft_consensus.cc:2229] T 67dbaa28babb4f64810d45c87198df0d P 484a1904825c40c882b646a103a92741 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:39:06.570143 27402 raft_consensus.cc:2258] T 67dbaa28babb4f64810d45c87198df0d P 484a1904825c40c882b646a103a92741 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:39:06.571707 27403 tablet_replica.cc:330] T 67dbaa28babb4f64810d45c87198df0d P 4bfa38ed7208453cb9bf7785e446e0a3: stopping tablet replica
I20260924 11:39:06.572187 27403 raft_consensus.cc:2229] T 67dbaa28babb4f64810d45c87198df0d P 4bfa38ed7208453cb9bf7785e446e0a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 11:39:06.572721 27403 raft_consensus.cc:2258] T 67dbaa28babb4f64810d45c87198df0d P 4bfa38ed7208453cb9bf7785e446e0a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:39:06.573571 27401 tablet_replica.cc:330] T 67dbaa28babb4f64810d45c87198df0d P 0993799c1b0343adac74fe5e5e52bec3: stopping tablet replica
I20260924 11:39:06.574012 27401 raft_consensus.cc:2229] T 67dbaa28babb4f64810d45c87198df0d P 0993799c1b0343adac74fe5e5e52bec3 [term 1 LEADER]: Raft consensus shutting down.
I20260924 11:39:06.574790 27401 raft_consensus.cc:2258] T 67dbaa28babb4f64810d45c87198df0d P 0993799c1b0343adac74fe5e5e52bec3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 11:39:06.575946 27403 ts_tablet_manager.cc:1905] T 67dbaa28babb4f64810d45c87198df0d P 4bfa38ed7208453cb9bf7785e446e0a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:39:06.576860 27401 ts_tablet_manager.cc:1905] T 67dbaa28babb4f64810d45c87198df0d P 0993799c1b0343adac74fe5e5e52bec3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:39:06.581328 27375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:06.560862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34638 (local address 127.6.212.1:45647)
0924 11:39:06.561108 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:06.561118 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:06.570962 (+ 9844us) server_negotiation.cc:407] Connection header received
0924 11:39:06.571228 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:06.571236 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:06.571453 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:06.571567 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:06.574069 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:06.575237 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:06.577152 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:06.577819 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:06.577929 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:06.579445 (+ 1516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:06.579456 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:06.579475 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:39:06.580351 (+ 876us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:06.580419 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:39:06.580442 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:06.580519 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:06.580783 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 11:39:06.580964 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 11:39:06.582508 27092 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:39:06.584434 27403 ts_tablet_manager.cc:1918] T 67dbaa28babb4f64810d45c87198df0d P 4bfa38ed7208453cb9bf7785e446e0a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:39:06.584762 27403 log.cc:1196] T 67dbaa28babb4f64810d45c87198df0d P 4bfa38ed7208453cb9bf7785e446e0a3: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-1/wal/wals/67dbaa28babb4f64810d45c87198df0d
I20260924 11:39:06.584812 27402 ts_tablet_manager.cc:1905] T 67dbaa28babb4f64810d45c87198df0d P 484a1904825c40c882b646a103a92741: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 11:39:06.585551 27403 ts_tablet_manager.cc:1939] T 67dbaa28babb4f64810d45c87198df0d P 4bfa38ed7208453cb9bf7785e446e0a3: Deleting consensus metadata
I20260924 11:39:06.587961 26897 catalog_manager.cc:4841] TS 4bfa38ed7208453cb9bf7785e446e0a3 (127.6.212.2:37335): tablet 67dbaa28babb4f64810d45c87198df0d (table test-table [id=8264f64386f345e09418003c14f4822d]) successfully deleted
I20260924 11:39:06.593245 27402 ts_tablet_manager.cc:1918] T 67dbaa28babb4f64810d45c87198df0d P 484a1904825c40c882b646a103a92741: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 11:39:06.593428 27402 log.cc:1196] T 67dbaa28babb4f64810d45c87198df0d P 484a1904825c40c882b646a103a92741: Deleting WAL directory at /tmp/dist-test-task75Snja/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790249617057452-6992-0/minicluster-data/ts-2/wal/wals/67dbaa28babb4f64810d45c87198df0d
I20260924 11:39:06.593883 27402 ts_tablet_manager.cc:1939] T 67dbaa28babb4f64810d45c87198df0d P 484a1904825c40c882b646a103a92741: Deleting consensus metadata
I20260924 11:39:06.595642 26898 catalog_manager.cc:4841] TS 484a1904825c40c882b646a103a92741 (127.6.212.3:43963): tablet 67dbaa28babb4f64810d45c87198df0d (table test-table [id=8264f64386f345e09418003c14f4822d]) successfully deleted
I20260924 11:39:06.686201 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 26970
I20260924 11:39:06.704562 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 27101
W20260924 11:39:06.704591 26898 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.212.1:45647 (error 108)
I20260924 11:39:06.706472 27371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:06.705898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:45647 (local address 127.0.0.1:34640)
0924 11:39:06.706126 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:06.706285 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.1:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
W20260924 11:39:06.706774 26898 catalog_manager.cc:4584] TS 0993799c1b0343adac74fe5e5e52bec3 (127.6.212.1:45647): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 67dbaa28babb4f64810d45c87198df0d: Network error: Client connection negotiation failed: client connection to 127.6.212.1:45647: connect: Connection refused (error 111)
I20260924 11:39:06.711912 27388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:06.705362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56466 (local address 127.6.212.2:37335)
0924 11:39:06.705542 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:06.705546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:06.705705 (+ 159us) server_negotiation.cc:407] Connection header received
0924 11:39:06.705945 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:06.705951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:06.706114 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:06.706202 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:06.706785 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:06.707566 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:06.708786 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:06.709289 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:06.709410 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:06.710451 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:06.710459 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:06.710473 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:39:06.711042 (+ 569us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:06.711087 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:39:06.711106 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:06.711193 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:06.711345 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 11:39:06.711525 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 11:39:06.712540 27223 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:39:06.760524 27371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:06.760018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:45647 (local address 127.0.0.1:34648)
0924 11:39:06.760278 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:06.760372 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.1:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260924 11:39:06.761674 27371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:06.761319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:45647 (local address 127.0.0.1:34654)
0924 11:39:06.761477 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:06.761553 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.1:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 11:39:06.805503 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 27101
I20260924 11:39:06.822294 27371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:06.821649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:45647 (local address 127.0.0.1:34664)
0924 11:39:06.821923 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:06.822070 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.1:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260924 11:39:06.823621 27371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:06.823082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:45647 (local address 127.0.0.1:34674)
0924 11:39:06.823308 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:06.823420 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.1:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260924 11:39:06.823704 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 27231
I20260924 11:39:06.838044 27374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:06.824504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53714 (local address 127.6.212.3:43963)
0924 11:39:06.824655 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:06.824660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:06.824677 (+ 17us) server_negotiation.cc:407] Connection header received
0924 11:39:06.824818 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:06.824822 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:06.824948 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:06.825033 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:06.825433 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:06.826191 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:06.827590 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:06.828032 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:06.828146 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:06.836526 (+ 8380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:06.836542 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:06.836566 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:39:06.837193 (+ 627us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:06.837270 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:39:06.837321 (+ 51us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:06.837456 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:06.837630 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 11:39:06.837742 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 11:39:06.838757 27353 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:39:06.904539 27371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:06.903950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:45647 (local address 127.0.0.1:34688)
0924 11:39:06.904232 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:06.904349 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.1:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260924 11:39:06.905814 27371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:06.905312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.1:45647 (local address 127.0.0.1:34696)
0924 11:39:06.905539 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:06.905617 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.1:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260924 11:39:06.930397 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 27231
I20260924 11:39:06.948020 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 26880
I20260924 11:39:06.949277 26941 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:39:07.060967 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 26880
2026-09-24T11:39:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3004 ms)
[----------] 6 tests from EncryptionPolicyTest (22800 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790249617057452-6992-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:39:07 dist-test-slave-cq7c krb5kdc[27412](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:39:07 dist-test-slave-cq7c krb5kdc[27412](info): set up 2 sockets
Sep 24 11:39:07 dist-test-slave-cq7c krb5kdc[27412](info): commencing operation
krb5kdc: starting...
W20260924 11:39:08.175770 6992 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Sep 24 11:39:08 dist-test-slave-cq7c krb5kdc[27412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:39:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:39:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790249617057452-6992-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790249617057452-6992-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 11:39:08.448063 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41071
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790249617057452-6992-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:39:08.792387 27425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:39:08.792701 27425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:39:08.792862 27425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:39:08.800288 27425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:39:08.800418 27425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:39:08.800547 27425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:39:08.800680 27425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:39:08.800753 27425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:39:08.810964 27425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41071
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790249617057452-6992-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:39:08.812374 27425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:39:08.813773 27425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:39:08.826963 27430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:08.826921 27433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:08.826956 27431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:39:08.827911 27425 server_base.cc:1034] running on GCE node
I20260924 11:39:08.828810 27425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:39:08.830300 27425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:39:08.831472 27425 hybrid_clock.cc:648] HybridClock initialized: now 1790249948831439 us; error 29 us; skew 500 ppm
I20260924 11:39:08.831868 27425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:39:08 dist-test-slave-cq7c krb5kdc[27412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:39:08.835261 27425 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 11:39:08.836783 27425 webserver.cc:460] Webserver started at http://127.0.0.1:46381/ using document root <none> and password file <none>
I20260924 11:39:08.837520 27425 fs_manager.cc:359] Metadata directory not provided
I20260924 11:39:08.837666 27425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:39:08.837968 27425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:39:08.841315 27425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "f8724e988d06497b98d26efe36eaca66"
format_stamp: "Formatted at 2026-09-24 11:39:08 on dist-test-slave-cq7c"
server_key: "7a9ca5a1107110c6e3ec018a458a9bd9"
server_key_iv: "8c4b3d5de35fa423c63c874beb556208"
server_key_version: "encryptionkey@0"
I20260924 11:39:08.842200 27425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "f8724e988d06497b98d26efe36eaca66"
format_stamp: "Formatted at 2026-09-24 11:39:08 on dist-test-slave-cq7c"
server_key: "7a9ca5a1107110c6e3ec018a458a9bd9"
server_key_iv: "8c4b3d5de35fa423c63c874beb556208"
server_key_version: "encryptionkey@0"
I20260924 11:39:08.848099 27425 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260924 11:39:08.851750 27440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:39:08.853320 27425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260924 11:39:08.853536 27425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "f8724e988d06497b98d26efe36eaca66"
format_stamp: "Formatted at 2026-09-24 11:39:08 on dist-test-slave-cq7c"
server_key: "7a9ca5a1107110c6e3ec018a458a9bd9"
server_key_iv: "8c4b3d5de35fa423c63c874beb556208"
server_key_version: "encryptionkey@0"
I20260924 11:39:08.853818 27425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:39:08.873806 27425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:39:08.878211 27425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:39:08.878530 27425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:39:08.898635 27425 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41071
I20260924 11:39:08.898659 27491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41071 every 8 connection(s)
I20260924 11:39:08.900250 27425 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:39:08.904301 27492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:39:08.904699 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 27425
I20260924 11:39:08.904925 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:39:08.905722 6992 external_mini_cluster.cc:1428] Setting key 50b68f8b3a5b3aecc9c62ba06fa0b1f3
I20260924 11:39:08.918922 27492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8724e988d06497b98d26efe36eaca66: Bootstrap starting.
Sep 24 11:39:08 dist-test-slave-cq7c krb5kdc[27412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 11:39:08.922704 27492 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8724e988d06497b98d26efe36eaca66: Neither blocks nor log segments found. Creating new log.
I20260924 11:39:08.923926 27492 log.cc:824] T 00000000000000000000000000000000 P f8724e988d06497b98d26efe36eaca66: Log is configured to *not* fsync() on all Append() calls
I20260924 11:39:08.926805 27492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8724e988d06497b98d26efe36eaca66: No bootstrap required, opened a new log
I20260924 11:39:08.930282 27495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:08.907713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40289 (local address 127.0.0.1:41071)
0924 11:39:08.908579 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:08.908591 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:08.908635 (+ 44us) server_negotiation.cc:407] Connection header received
0924 11:39:08.910722 (+ 2087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:08.910755 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:08.911173 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:08.911602 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:08.913488 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:08.914887 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:08.916736 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:08.917391 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:08.917529 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:08.922332 (+ 4803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:08.922340 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:08.922356 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:39:08.922391 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:08.925631 (+ 3240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:08.926492 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:08.926498 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:08.926504 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:08.926660 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:08.927430 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:08.927440 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:08.927443 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:08.927972 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:08.928350 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:08.928860 (+ 510us) server_negotiation.cc:299] Negotiation successful
0924 11:39:08.929199 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":417,"threads_started":1}
I20260924 11:39:08.934379 6992 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:39:08.935779 27492 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8724e988d06497b98d26efe36eaca66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8724e988d06497b98d26efe36eaca66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41071 } }
I20260924 11:39:08.936195 27492 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8724e988d06497b98d26efe36eaca66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:39:08.936304 27492 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8724e988d06497b98d26efe36eaca66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8724e988d06497b98d26efe36eaca66, State: Initialized, Role: FOLLOWER
I20260924 11:39:08.937008 27492 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8724e988d06497b98d26efe36eaca66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8724e988d06497b98d26efe36eaca66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41071 } }
I20260924 11:39:08.937319 27492 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8724e988d06497b98d26efe36eaca66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:39:08.937485 27492 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8724e988d06497b98d26efe36eaca66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:39:08.937664 27492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8724e988d06497b98d26efe36eaca66 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:39:08.940006 27492 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8724e988d06497b98d26efe36eaca66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8724e988d06497b98d26efe36eaca66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41071 } }
I20260924 11:39:08.940667 27492 leader_election.cc:302] T 00000000000000000000000000000000 P f8724e988d06497b98d26efe36eaca66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8724e988d06497b98d26efe36eaca66; no voters:
I20260924 11:39:08.941504 27492 leader_election.cc:288] T 00000000000000000000000000000000 P f8724e988d06497b98d26efe36eaca66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:39:08.942530 27502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8724e988d06497b98d26efe36eaca66 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:39:08.943310 27502 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8724e988d06497b98d26efe36eaca66 [term 1 LEADER]: Becoming Leader. State: Replica: f8724e988d06497b98d26efe36eaca66, State: Running, Role: LEADER
I20260924 11:39:08.943991 27502 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8724e988d06497b98d26efe36eaca66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8724e988d06497b98d26efe36eaca66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41071 } }
I20260924 11:39:08.944155 27492 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8724e988d06497b98d26efe36eaca66 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:39:08.953423 27504 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8724e988d06497b98d26efe36eaca66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8724e988d06497b98d26efe36eaca66. Latest consensus state: current_term: 1 leader_uuid: "f8724e988d06497b98d26efe36eaca66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8724e988d06497b98d26efe36eaca66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41071 } } }
I20260924 11:39:08.953825 27504 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8724e988d06497b98d26efe36eaca66 [sys.catalog]: This master's current role is: LEADER
I20260924 11:39:08.954232 27503 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8724e988d06497b98d26efe36eaca66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8724e988d06497b98d26efe36eaca66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8724e988d06497b98d26efe36eaca66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41071 } } }
I20260924 11:39:08.954605 27503 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8724e988d06497b98d26efe36eaca66 [sys.catalog]: This master's current role is: LEADER
I20260924 11:39:08.957479 27515 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:39:08.960311 27495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:08.940342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45369 (local address 127.0.0.1:41071)
0924 11:39:08.940536 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:08.940543 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:08.941014 (+ 471us) server_negotiation.cc:407] Connection header received
0924 11:39:08.941148 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:08.941157 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:08.941268 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:08.941452 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:08.943749 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:08.944921 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:08.947273 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:08.948210 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:08.948411 (+ 201us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:08.952371 (+ 3960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:08.952378 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:08.952382 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:39:08.952421 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:08.956877 (+ 4456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:08.957819 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:08.957825 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:08.957828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:08.957986 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:08.958531 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:08.958534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:08.958537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:08.959041 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:08.959343 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:08.959761 (+ 418us) server_negotiation.cc:299] Negotiation successful
0924 11:39:08.959943 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 11:39:08.964828 27515 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:39:08.974925 27515 catalog_manager.cc:1347] Generated new cluster ID: 8a6b867c66ee47729afb76a1a87ad23b
I20260924 11:39:08.975113 27515 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:39:09.003430 27515 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:39:09.005012 27515 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:39:09.015182 27515 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8724e988d06497b98d26efe36eaca66: Generated new TSK 0
I20260924 11:39:09.016217 27515 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:39:09.029992 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 27425
I20260924 11:39:09.031477 27487 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:39:09.125789 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 27425
W20260924 11:39:09.146404 27410 connection.cc:535] client connection to 127.0.0.1:41071 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41071): Connection reset by peer (error 104)
2026-09-24T11:39:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2060 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790249617057452-6992-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:39:09 dist-test-slave-cq7c krb5kdc[27524](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:39:09 dist-test-slave-cq7c krb5kdc[27524](info): set up 2 sockets
Sep 24 11:39:09 dist-test-slave-cq7c krb5kdc[27524](info): commencing operation
krb5kdc: starting...
W20260924 11:39:10.259797 6992 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.002s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Sep 24 11:39:10 dist-test-slave-cq7c krb5kdc[27524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:39:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:39:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790249617057452-6992-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790249617057452-6992-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 11:39:10.539841 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36237
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790249617057452-6992-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36237
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:39:10.869342 27537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:39:10.869625 27537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:39:10.869802 27537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:39:10.876926 27537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:39:10.877036 27537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:39:10.877148 27537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:39:10.877251 27537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:39:10.877333 27537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:39:10.887444 27537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36237
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790249617057452-6992-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:39:10.888901 27537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:39:10.890213 27537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:39:10.902456 27542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:10.902357 27543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:10.902397 27545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:39:10.903255 27537 server_base.cc:1034] running on GCE node
I20260924 11:39:10.904160 27537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:39:10.905560 27537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:39:10.906816 27537 hybrid_clock.cc:648] HybridClock initialized: now 1790249950906756 us; error 52 us; skew 500 ppm
I20260924 11:39:10.907182 27537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:39:10 dist-test-slave-cq7c krb5kdc[27524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249950, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:39:10.916828 27537 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 11:39:10.918394 27537 webserver.cc:460] Webserver started at http://127.0.0.1:33999/ using document root <none> and password file <none>
I20260924 11:39:10.919149 27537 fs_manager.cc:359] Metadata directory not provided
I20260924 11:39:10.919299 27537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:39:10.919695 27537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:39:10.922770 27537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "70b9209ede7b4c5db470ade9b940b31a"
format_stamp: "Formatted at 2026-09-24 11:39:10 on dist-test-slave-cq7c"
server_key: "cf061d564e974a8114b7266bb189005f"
server_key_iv: "201cae23f6c3452f2bf3a607a9cca709"
server_key_version: "encryptionkey@0"
I20260924 11:39:10.923592 27537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "70b9209ede7b4c5db470ade9b940b31a"
format_stamp: "Formatted at 2026-09-24 11:39:10 on dist-test-slave-cq7c"
server_key: "cf061d564e974a8114b7266bb189005f"
server_key_iv: "201cae23f6c3452f2bf3a607a9cca709"
server_key_version: "encryptionkey@0"
I20260924 11:39:10.929121 27537 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 11:39:10.933032 27552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:39:10.934532 27537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260924 11:39:10.934749 27537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "70b9209ede7b4c5db470ade9b940b31a"
format_stamp: "Formatted at 2026-09-24 11:39:10 on dist-test-slave-cq7c"
server_key: "cf061d564e974a8114b7266bb189005f"
server_key_iv: "201cae23f6c3452f2bf3a607a9cca709"
server_key_version: "encryptionkey@0"
I20260924 11:39:10.934962 27537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:39:10.956712 27537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:39:10.961282 27537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:39:10.961576 27537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:39:10.980563 27537 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36237
I20260924 11:39:10.980595 27603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36237 every 8 connection(s)
I20260924 11:39:10.981977 27537 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:39:10.985754 27604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:39:10.986586 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 27537
I20260924 11:39:10.986771 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:39:10.987339 6992 external_mini_cluster.cc:1428] Setting key e52c377c64bd60ab3e9d0c419ba32a75
I20260924 11:39:10.998486 27604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70b9209ede7b4c5db470ade9b940b31a: Bootstrap starting.
I20260924 11:39:11.000801 27607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:10.989368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39103 (local address 127.0.0.1:36237)
0924 11:39:10.990209 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:10.990221 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:10.990261 (+ 40us) server_negotiation.cc:407] Connection header received
0924 11:39:10.992329 (+ 2068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:10.992353 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:10.992799 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:10.993198 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:10.993733 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:10.994992 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:10.996385 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:10.997058 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:10.997265 (+ 207us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:10.998518 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:10.998525 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:10.998539 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:39:10.999118 (+ 579us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:10.999194 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:39:10.999226 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:10.999419 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:10.999696 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 11:39:10.999861 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":361,"threads_started":1}
I20260924 11:39:11.001793 27604 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70b9209ede7b4c5db470ade9b940b31a: Neither blocks nor log segments found. Creating new log.
I20260924 11:39:11.002905 27604 log.cc:824] T 00000000000000000000000000000000 P 70b9209ede7b4c5db470ade9b940b31a: Log is configured to *not* fsync() on all Append() calls
I20260924 11:39:11.004611 6992 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:39:11.006276 27604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70b9209ede7b4c5db470ade9b940b31a: No bootstrap required, opened a new log
I20260924 11:39:11.019553 27604 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70b9209ede7b4c5db470ade9b940b31a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70b9209ede7b4c5db470ade9b940b31a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36237 } }
I20260924 11:39:11.020072 27604 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70b9209ede7b4c5db470ade9b940b31a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:39:11.020201 27604 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70b9209ede7b4c5db470ade9b940b31a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70b9209ede7b4c5db470ade9b940b31a, State: Initialized, Role: FOLLOWER
I20260924 11:39:11.021111 27604 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70b9209ede7b4c5db470ade9b940b31a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70b9209ede7b4c5db470ade9b940b31a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36237 } }
I20260924 11:39:11.021399 27604 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70b9209ede7b4c5db470ade9b940b31a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:39:11.021552 27604 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70b9209ede7b4c5db470ade9b940b31a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:39:11.021739 27604 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70b9209ede7b4c5db470ade9b940b31a [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:39:11.023733 27607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:11.011712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41307 (local address 127.0.0.1:36237)
0924 11:39:11.011898 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:11.011904 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:11.015469 (+ 3565us) server_negotiation.cc:407] Connection header received
0924 11:39:11.015561 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:11.015565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:11.015630 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:11.015740 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:11.016372 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:11.017796 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:11.020159 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:11.021013 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:11.021141 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:11.022679 (+ 1538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:11.022686 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:11.022692 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:39:11.022713 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:11.022771 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:39:11.022795 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:11.022909 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:11.023161 (+ 252us) server_negotiation.cc:299] Negotiation successful
0924 11:39:11.023226 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 11:39:11.025180 27604 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70b9209ede7b4c5db470ade9b940b31a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70b9209ede7b4c5db470ade9b940b31a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36237 } }
I20260924 11:39:11.025795 27604 leader_election.cc:302] T 00000000000000000000000000000000 P 70b9209ede7b4c5db470ade9b940b31a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70b9209ede7b4c5db470ade9b940b31a; no voters:
I20260924 11:39:11.026521 27604 leader_election.cc:288] T 00000000000000000000000000000000 P 70b9209ede7b4c5db470ade9b940b31a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:39:11.026639 27614 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70b9209ede7b4c5db470ade9b940b31a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:39:11.027400 27614 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70b9209ede7b4c5db470ade9b940b31a [term 1 LEADER]: Becoming Leader. State: Replica: 70b9209ede7b4c5db470ade9b940b31a, State: Running, Role: LEADER
I20260924 11:39:11.028000 27614 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70b9209ede7b4c5db470ade9b940b31a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70b9209ede7b4c5db470ade9b940b31a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36237 } }
I20260924 11:39:11.028683 27604 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70b9209ede7b4c5db470ade9b940b31a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:39:11.030578 27616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70b9209ede7b4c5db470ade9b940b31a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70b9209ede7b4c5db470ade9b940b31a. Latest consensus state: current_term: 1 leader_uuid: "70b9209ede7b4c5db470ade9b940b31a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70b9209ede7b4c5db470ade9b940b31a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36237 } } }
I20260924 11:39:11.031575 27616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70b9209ede7b4c5db470ade9b940b31a [sys.catalog]: This master's current role is: LEADER
I20260924 11:39:11.031170 27615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70b9209ede7b4c5db470ade9b940b31a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70b9209ede7b4c5db470ade9b940b31a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70b9209ede7b4c5db470ade9b940b31a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36237 } } }
I20260924 11:39:11.032096 27615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70b9209ede7b4c5db470ade9b940b31a [sys.catalog]: This master's current role is: LEADER
I20260924 11:39:11.036073 27620 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:39:11.043783 27620 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:39:11.062664 27620 catalog_manager.cc:1347] Generated new cluster ID: 6ddaeaa52bbe439e958a1c7729651317
I20260924 11:39:11.062793 27620 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:39:11.084051 27620 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:39:11.085945 27620 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:39:11.096026 27620 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70b9209ede7b4c5db470ade9b940b31a: Generated new TSK 0
I20260924 11:39:11.096946 27620 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:39:11.113029 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 27537
W20260924 11:39:11.114228 27599 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39103
W20260924 11:39:11.114964 6992 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 27537: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:39:11.115154 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 27537
W20260924 11:39:11.132871 27522 connection.cc:535] client connection to 127.0.0.1:36237 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36237): Connection reset by peer (error 104)
2026-09-24T11:39:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1986 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790249617057452-6992-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:39:11 dist-test-slave-cq7c krb5kdc[27635](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:39:11 dist-test-slave-cq7c krb5kdc[27635](info): set up 2 sockets
Sep 24 11:39:11 dist-test-slave-cq7c krb5kdc[27635](info): commencing operation
krb5kdc: starting...
W20260924 11:39:12.247882 6992 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.002s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Sep 24 11:39:12 dist-test-slave-cq7c krb5kdc[27635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:39:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:39:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790249617057452-6992-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790249617057452-6992-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 11:39:12.550997 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46723
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790249617057452-6992-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:39:12.891548 27648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:39:12.891855 27648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:39:12.892037 27648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:39:12.899472 27648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:39:12.899605 27648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:39:12.899695 27648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:39:12.899775 27648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:39:12.899859 27648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:39:12.909891 27648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46723
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790249617057452-6992-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:39:12.911207 27648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:39:12.912566 27648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:39:12.924551 27654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:12.924592 27656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:12.924636 27653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:39:12.925509 27648 server_base.cc:1034] running on GCE node
I20260924 11:39:12.926338 27648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:39:12.927865 27648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:39:12.929109 27648 hybrid_clock.cc:648] HybridClock initialized: now 1790249952929059 us; error 37 us; skew 500 ppm
I20260924 11:39:12.929466 27648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:39:12 dist-test-slave-cq7c krb5kdc[27635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:39:12.940279 27648 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 11:39:12.941995 27648 webserver.cc:460] Webserver started at http://127.0.0.1:38369/ using document root <none> and password file <none>
I20260924 11:39:12.942764 27648 fs_manager.cc:359] Metadata directory not provided
I20260924 11:39:12.942924 27648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:39:12.943259 27648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:39:12.946374 27648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "6f430b30440245aaa1b47f6a9b365631"
format_stamp: "Formatted at 2026-09-24 11:39:12 on dist-test-slave-cq7c"
server_key: "e425798c2c9ca4865bbc8e26152a75c5"
server_key_iv: "ee946a543ebb3f375308da1b79fb5728"
server_key_version: "encryptionkey@0"
I20260924 11:39:12.947160 27648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "6f430b30440245aaa1b47f6a9b365631"
format_stamp: "Formatted at 2026-09-24 11:39:12 on dist-test-slave-cq7c"
server_key: "e425798c2c9ca4865bbc8e26152a75c5"
server_key_iv: "ee946a543ebb3f375308da1b79fb5728"
server_key_version: "encryptionkey@0"
I20260924 11:39:12.952785 27648 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 11:39:12.956339 27663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:39:12.957913 27648 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 11:39:12.958112 27648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "6f430b30440245aaa1b47f6a9b365631"
format_stamp: "Formatted at 2026-09-24 11:39:12 on dist-test-slave-cq7c"
server_key: "e425798c2c9ca4865bbc8e26152a75c5"
server_key_iv: "ee946a543ebb3f375308da1b79fb5728"
server_key_version: "encryptionkey@0"
I20260924 11:39:12.958330 27648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:39:12.964536 27648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:39:12.968638 27648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:39:12.968935 27648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:39:12.987493 27648 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46723
I20260924 11:39:12.987637 27714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46723 every 8 connection(s)
I20260924 11:39:12.989126 27648 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:39:12.993047 27715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:39:12.998914 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 27648
I20260924 11:39:12.999089 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:39:12.999596 6992 external_mini_cluster.cc:1428] Setting key ce0f53a606b68eac7196a40c3f005fef
I20260924 11:39:13.004098 27715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f430b30440245aaa1b47f6a9b365631: Bootstrap starting.
I20260924 11:39:13.007658 27718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:13.001494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36159 (local address 127.0.0.1:46723)
0924 11:39:13.002363 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:13.002375 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:13.002422 (+ 47us) server_negotiation.cc:407] Connection header received
0924 11:39:13.004116 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:13.004139 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:13.004540 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:13.004876 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:13.005993 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:13.005999 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:13.006011 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:39:13.006024 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:13.006084 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:39:13.006107 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:13.006265 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:13.006430 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 11:39:13.006625 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":407,"threads_started":1}
I20260924 11:39:13.007998 27715 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f430b30440245aaa1b47f6a9b365631: Neither blocks nor log segments found. Creating new log.
I20260924 11:39:13.009351 27715 log.cc:824] T 00000000000000000000000000000000 P 6f430b30440245aaa1b47f6a9b365631: Log is configured to *not* fsync() on all Append() calls
I20260924 11:39:13.011742 6992 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:39:13.012754 27715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f430b30440245aaa1b47f6a9b365631: No bootstrap required, opened a new log
I20260924 11:39:13.019577 27718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:13.016887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54621 (local address 127.0.0.1:46723)
0924 11:39:13.017106 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:13.017115 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:13.017608 (+ 493us) server_negotiation.cc:407] Connection header received
0924 11:39:13.017790 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:13.017794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:13.017842 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:13.017923 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:13.018887 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:13.018891 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:13.018895 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:39:13.018909 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:13.018954 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:39:13.018969 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:13.019068 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:13.019193 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 11:39:13.019249 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 11:39:13.022655 27715 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f430b30440245aaa1b47f6a9b365631 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f430b30440245aaa1b47f6a9b365631" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46723 } }
I20260924 11:39:13.023061 27715 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f430b30440245aaa1b47f6a9b365631 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:39:13.023185 27715 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f430b30440245aaa1b47f6a9b365631 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f430b30440245aaa1b47f6a9b365631, State: Initialized, Role: FOLLOWER
I20260924 11:39:13.023821 27715 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f430b30440245aaa1b47f6a9b365631 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f430b30440245aaa1b47f6a9b365631" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46723 } }
I20260924 11:39:13.024065 27715 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f430b30440245aaa1b47f6a9b365631 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:39:13.024219 27715 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f430b30440245aaa1b47f6a9b365631 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:39:13.024398 27715 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f430b30440245aaa1b47f6a9b365631 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:39:13.026654 27715 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f430b30440245aaa1b47f6a9b365631 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f430b30440245aaa1b47f6a9b365631" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46723 } }
I20260924 11:39:13.027210 27715 leader_election.cc:302] T 00000000000000000000000000000000 P 6f430b30440245aaa1b47f6a9b365631 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f430b30440245aaa1b47f6a9b365631; no voters:
I20260924 11:39:13.027871 27715 leader_election.cc:288] T 00000000000000000000000000000000 P 6f430b30440245aaa1b47f6a9b365631 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:39:13.028021 27725 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f430b30440245aaa1b47f6a9b365631 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:39:13.028683 27725 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f430b30440245aaa1b47f6a9b365631 [term 1 LEADER]: Becoming Leader. State: Replica: 6f430b30440245aaa1b47f6a9b365631, State: Running, Role: LEADER
I20260924 11:39:13.029201 27725 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f430b30440245aaa1b47f6a9b365631 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f430b30440245aaa1b47f6a9b365631" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46723 } }
I20260924 11:39:13.029963 27715 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f430b30440245aaa1b47f6a9b365631 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:39:13.032541 27727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f430b30440245aaa1b47f6a9b365631 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f430b30440245aaa1b47f6a9b365631. Latest consensus state: current_term: 1 leader_uuid: "6f430b30440245aaa1b47f6a9b365631" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f430b30440245aaa1b47f6a9b365631" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46723 } } }
I20260924 11:39:13.032946 27727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f430b30440245aaa1b47f6a9b365631 [sys.catalog]: This master's current role is: LEADER
I20260924 11:39:13.033313 27726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f430b30440245aaa1b47f6a9b365631 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f430b30440245aaa1b47f6a9b365631" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f430b30440245aaa1b47f6a9b365631" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46723 } } }
I20260924 11:39:13.033648 27726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f430b30440245aaa1b47f6a9b365631 [sys.catalog]: This master's current role is: LEADER
I20260924 11:39:13.034767 27731 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:39:13.039955 27731 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:39:13.048740 27731 catalog_manager.cc:1347] Generated new cluster ID: d3341aca443743cb81702fa874dc775b
I20260924 11:39:13.048902 27731 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:39:13.094324 27731 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:39:13.094852 27731 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:39:13.102802 27731 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f430b30440245aaa1b47f6a9b365631: Generated new TSK 0
I20260924 11:39:13.103826 27731 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:39:13.117380 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 27648
W20260924 11:39:13.119410 27710 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36159
W20260924 11:39:13.120093 6992 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 27648: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:39:13.120263 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 27648
2026-09-24T11:39:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2004 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790249617057452-6992-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:39:13 dist-test-slave-cq7c krb5kdc[27746](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:39:13 dist-test-slave-cq7c krb5kdc[27746](info): set up 2 sockets
Sep 24 11:39:13 dist-test-slave-cq7c krb5kdc[27746](info): commencing operation
krb5kdc: starting...
W20260924 11:39:14.243909 6992 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Sep 24 11:39:14 dist-test-slave-cq7c krb5kdc[27746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:39:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:39:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790249617057452-6992-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790249617057452-6992-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 11:39:14.531026 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41213
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790249617057452-6992-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:39:14.885735 27759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:39:14.886072 27759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:39:14.886183 27759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:39:14.894439 27759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:39:14.894572 27759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:39:14.894618 27759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:39:14.894680 27759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:39:14.894724 27759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:39:14.905061 27759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790249617057452-6992-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:39:14.906381 27759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:39:14.907821 27759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:39:14.919826 27764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:14.919945 27765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:14.919914 27767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:39:14.920712 27759 server_base.cc:1034] running on GCE node
I20260924 11:39:14.921558 27759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:39:14.923018 27759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:39:14.924263 27759 hybrid_clock.cc:648] HybridClock initialized: now 1790249954924216 us; error 42 us; skew 500 ppm
I20260924 11:39:14.924630 27759 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:39:14 dist-test-slave-cq7c krb5kdc[27746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:39:14.927958 27759 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 11:39:14.929574 27759 webserver.cc:460] Webserver started at http://127.0.0.1:40395/ using document root <none> and password file <none>
I20260924 11:39:14.930294 27759 fs_manager.cc:359] Metadata directory not provided
I20260924 11:39:14.930438 27759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:39:14.930749 27759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:39:14.933842 27759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "88993c06508f456eb61feaad8fe06da7"
format_stamp: "Formatted at 2026-09-24 11:39:14 on dist-test-slave-cq7c"
server_key: "85738769aed952dcadbfe96963bd8f66"
server_key_iv: "66a8dfc5c3074f9e1f4aaff962431c75"
server_key_version: "encryptionkey@0"
I20260924 11:39:14.934732 27759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "88993c06508f456eb61feaad8fe06da7"
format_stamp: "Formatted at 2026-09-24 11:39:14 on dist-test-slave-cq7c"
server_key: "85738769aed952dcadbfe96963bd8f66"
server_key_iv: "66a8dfc5c3074f9e1f4aaff962431c75"
server_key_version: "encryptionkey@0"
I20260924 11:39:14.940114 27759 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 11:39:14.944176 27774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:39:14.945586 27759 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:39:14.945793 27759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "88993c06508f456eb61feaad8fe06da7"
format_stamp: "Formatted at 2026-09-24 11:39:14 on dist-test-slave-cq7c"
server_key: "85738769aed952dcadbfe96963bd8f66"
server_key_iv: "66a8dfc5c3074f9e1f4aaff962431c75"
server_key_version: "encryptionkey@0"
I20260924 11:39:14.946005 27759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:39:14.964133 27759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:39:14.968485 27759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:39:14.968779 27759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:39:14.988328 27759 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41213
I20260924 11:39:14.988339 27825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41213 every 8 connection(s)
I20260924 11:39:14.989758 27759 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:39:14.993667 27826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:39:14.998839 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 27759
I20260924 11:39:14.999030 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:39:14.999552 6992 external_mini_cluster.cc:1428] Setting key af59ad4384f378f68795c3434997a54c
I20260924 11:39:15.004859 27826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88993c06508f456eb61feaad8fe06da7: Bootstrap starting.
I20260924 11:39:15.008091 27826 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88993c06508f456eb61feaad8fe06da7: Neither blocks nor log segments found. Creating new log.
I20260924 11:39:15.009133 27826 log.cc:824] T 00000000000000000000000000000000 P 88993c06508f456eb61feaad8fe06da7: Log is configured to *not* fsync() on all Append() calls
I20260924 11:39:15.012810 27826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88993c06508f456eb61feaad8fe06da7: No bootstrap required, opened a new log
Sep 24 11:39:15 dist-test-slave-cq7c krb5kdc[27746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 11:39:15.021095 27826 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88993c06508f456eb61feaad8fe06da7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88993c06508f456eb61feaad8fe06da7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41213 } }
I20260924 11:39:15.021538 27826 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88993c06508f456eb61feaad8fe06da7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:39:15.021663 27826 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88993c06508f456eb61feaad8fe06da7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88993c06508f456eb61feaad8fe06da7, State: Initialized, Role: FOLLOWER
I20260924 11:39:15.022395 27826 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88993c06508f456eb61feaad8fe06da7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88993c06508f456eb61feaad8fe06da7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41213 } }
I20260924 11:39:15.022710 27826 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88993c06508f456eb61feaad8fe06da7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:39:15.022867 27826 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88993c06508f456eb61feaad8fe06da7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:39:15.023065 27826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88993c06508f456eb61feaad8fe06da7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:39:15.024645 27829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:15.001419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45907 (local address 127.0.0.1:41213)
0924 11:39:15.002253 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:15.002268 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:15.002310 (+ 42us) server_negotiation.cc:407] Connection header received
0924 11:39:15.004113 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:15.004136 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:15.004539 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:15.004909 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:15.006353 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:15.007742 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:15.009220 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:15.010316 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:15.010516 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:15.015743 (+ 5227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:15.015754 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:15.015772 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:39:15.015822 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:15.019615 (+ 3793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:15.020468 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:15.020476 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:15.020484 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:15.020691 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:15.021153 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:15.021161 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:15.021165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:15.021982 (+ 817us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:15.022288 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:15.022862 (+ 574us) server_negotiation.cc:299] Negotiation successful
0924 11:39:15.023450 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":364,"threads_started":1}
I20260924 11:39:15.025662 27826 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88993c06508f456eb61feaad8fe06da7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88993c06508f456eb61feaad8fe06da7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41213 } }
I20260924 11:39:15.026182 27826 leader_election.cc:302] T 00000000000000000000000000000000 P 88993c06508f456eb61feaad8fe06da7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88993c06508f456eb61feaad8fe06da7; no voters:
I20260924 11:39:15.026844 27826 leader_election.cc:288] T 00000000000000000000000000000000 P 88993c06508f456eb61feaad8fe06da7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:39:15.027016 27831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88993c06508f456eb61feaad8fe06da7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:39:15.028296 27831 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88993c06508f456eb61feaad8fe06da7 [term 1 LEADER]: Becoming Leader. State: Replica: 88993c06508f456eb61feaad8fe06da7, State: Running, Role: LEADER
I20260924 11:39:15.028961 27831 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88993c06508f456eb61feaad8fe06da7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88993c06508f456eb61feaad8fe06da7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41213 } }
I20260924 11:39:15.029246 6992 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:39:15.029353 27826 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88993c06508f456eb61feaad8fe06da7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:39:15.031876 27832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88993c06508f456eb61feaad8fe06da7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88993c06508f456eb61feaad8fe06da7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88993c06508f456eb61feaad8fe06da7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41213 } } }
I20260924 11:39:15.032306 27832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88993c06508f456eb61feaad8fe06da7 [sys.catalog]: This master's current role is: LEADER
I20260924 11:39:15.034868 27833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88993c06508f456eb61feaad8fe06da7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88993c06508f456eb61feaad8fe06da7. Latest consensus state: current_term: 1 leader_uuid: "88993c06508f456eb61feaad8fe06da7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88993c06508f456eb61feaad8fe06da7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41213 } } }
I20260924 11:39:15.035265 27833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88993c06508f456eb61feaad8fe06da7 [sys.catalog]: This master's current role is: LEADER
I20260924 11:39:15.037858 27836 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:39:15.043733 27836 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:39:15.070600 27836 catalog_manager.cc:1347] Generated new cluster ID: 355733f6c8af4bf3ab5a4eb559f8d3da
I20260924 11:39:15.070740 27836 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:39:15.074512 27829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:15.051431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54625 (local address 127.0.0.1:41213)
0924 11:39:15.051620 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:15.051626 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:15.052161 (+ 535us) server_negotiation.cc:407] Connection header received
0924 11:39:15.052487 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:15.052497 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:15.052605 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:15.052696 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:15.055354 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:15.056895 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:15.059039 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:15.060039 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:15.060190 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:15.064566 (+ 4376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:15.064572 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:15.064578 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:39:15.064627 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:15.069544 (+ 4917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:15.070773 (+ 1229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:15.070779 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:15.070784 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:15.070972 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:15.071615 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:15.071622 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:15.071628 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:15.072406 (+ 778us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:15.072711 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:15.073590 (+ 879us) server_negotiation.cc:299] Negotiation successful
0924 11:39:15.073933 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 11:39:15.080353 27836 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:39:15.082631 27836 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:39:15.098353 27836 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88993c06508f456eb61feaad8fe06da7: Generated new TSK 0
I20260924 11:39:15.099489 27836 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:39:15.105295 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 27759
I20260924 11:39:15.106901 27821 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:39:15.200169 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 27759
2026-09-24T11:39:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2080 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790249617057452-6992-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:39:15 dist-test-slave-cq7c krb5kdc[27858](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:39:15 dist-test-slave-cq7c krb5kdc[27858](info): set up 2 sockets
Sep 24 11:39:15 dist-test-slave-cq7c krb5kdc[27858](info): commencing operation
krb5kdc: starting...
W20260924 11:39:16.327986 6992 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.003s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Sep 24 11:39:16 dist-test-slave-cq7c krb5kdc[27858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:39:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:39:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790249617057452-6992-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790249617057452-6992-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 11:39:16.616737 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43627
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790249617057452-6992-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43627
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:39:16.962074 27871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:39:16.962441 27871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:39:16.962637 27871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:39:16.970094 27871 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:39:16.970252 27871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:39:16.970361 27871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:39:16.970465 27871 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:39:16.970552 27871 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:39:16.980633 27871 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43627
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790249617057452-6992-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:39:16.982069 27871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:39:16.983476 27871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:39:16.996701 27877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:16.996639 27879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:39:16.996902 27871 server_base.cc:1034] running on GCE node
W20260924 11:39:16.996724 27876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:39:16.998086 27871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:39:16.999657 27871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:39:17.000905 27871 hybrid_clock.cc:648] HybridClock initialized: now 1790249957000839 us; error 58 us; skew 500 ppm
I20260924 11:39:17.001272 27871 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:39:17 dist-test-slave-cq7c krb5kdc[27858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249957, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:39:17.004854 27871 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 11:39:17.006449 27871 webserver.cc:460] Webserver started at http://127.0.0.1:41675/ using document root <none> and password file <none>
I20260924 11:39:17.007184 27871 fs_manager.cc:359] Metadata directory not provided
I20260924 11:39:17.007335 27871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:39:17.007714 27871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:39:17.010753 27871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "d86f01415720495abfee7c5d7730d553"
format_stamp: "Formatted at 2026-09-24 11:39:17 on dist-test-slave-cq7c"
server_key: "6651e8af6d0bcf313a7db31120096b52"
server_key_iv: "81ee6ba5788103e67f9a4a9dd7ef7515"
server_key_version: "encryptionkey@0"
I20260924 11:39:17.011593 27871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "d86f01415720495abfee7c5d7730d553"
format_stamp: "Formatted at 2026-09-24 11:39:17 on dist-test-slave-cq7c"
server_key: "6651e8af6d0bcf313a7db31120096b52"
server_key_iv: "81ee6ba5788103e67f9a4a9dd7ef7515"
server_key_version: "encryptionkey@0"
I20260924 11:39:17.017261 27871 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 11:39:17.021114 27886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:39:17.022665 27871 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 11:39:17.022917 27871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "d86f01415720495abfee7c5d7730d553"
format_stamp: "Formatted at 2026-09-24 11:39:17 on dist-test-slave-cq7c"
server_key: "6651e8af6d0bcf313a7db31120096b52"
server_key_iv: "81ee6ba5788103e67f9a4a9dd7ef7515"
server_key_version: "encryptionkey@0"
I20260924 11:39:17.023134 27871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:39:17.035881 27871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:39:17.040436 27871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:39:17.040759 27871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:39:17.060972 27871 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43627
I20260924 11:39:17.061000 27937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43627 every 8 connection(s)
I20260924 11:39:17.062575 27871 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:39:17.064503 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 27871
I20260924 11:39:17.064739 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:39:17.065214 6992 external_mini_cluster.cc:1428] Setting key 4c7bc2854721e51b1057993b0a234178
I20260924 11:39:17.067376 27938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:39:17.080406 27941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:17.067087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37931 (local address 127.0.0.1:43627)
0924 11:39:17.068007 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:17.068020 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:17.068063 (+ 43us) server_negotiation.cc:407] Connection header received
0924 11:39:17.069849 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:17.069873 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:17.070316 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:17.070686 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:17.071331 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:17.072978 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:17.075121 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:17.075994 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:17.076173 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:17.077583 (+ 1410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:17.077591 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:17.077611 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:39:17.078260 (+ 649us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:17.078345 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:39:17.078380 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:17.078474 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:17.078784 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 11:39:17.079240 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":460,"threads_started":1}
I20260924 11:39:17.081835 27938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d86f01415720495abfee7c5d7730d553: Bootstrap starting.
I20260924 11:39:17.084787 6992 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:39:17.085858 27938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d86f01415720495abfee7c5d7730d553: Neither blocks nor log segments found. Creating new log.
I20260924 11:39:17.087260 27938 log.cc:824] T 00000000000000000000000000000000 P d86f01415720495abfee7c5d7730d553: Log is configured to *not* fsync() on all Append() calls
I20260924 11:39:17.090794 27938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d86f01415720495abfee7c5d7730d553: No bootstrap required, opened a new log
I20260924 11:39:17.097420 27941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:17.090442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46111 (local address 127.0.0.1:43627)
0924 11:39:17.090643 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:17.090649 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:17.091143 (+ 494us) server_negotiation.cc:407] Connection header received
0924 11:39:17.091329 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:17.091336 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:17.091453 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:17.091571 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:17.092094 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:17.093222 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:17.094638 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:17.095323 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:17.095485 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:17.096661 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:17.096668 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:17.096673 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:39:17.096696 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:17.096743 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:39:17.096763 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:17.096861 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:17.096987 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 11:39:17.097092 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 11:39:17.100243 27938 raft_consensus.cc:348] T 00000000000000000000000000000000 P d86f01415720495abfee7c5d7730d553 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d86f01415720495abfee7c5d7730d553" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43627 } }
I20260924 11:39:17.100663 27938 raft_consensus.cc:374] T 00000000000000000000000000000000 P d86f01415720495abfee7c5d7730d553 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:39:17.100787 27938 raft_consensus.cc:729] T 00000000000000000000000000000000 P d86f01415720495abfee7c5d7730d553 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d86f01415720495abfee7c5d7730d553, State: Initialized, Role: FOLLOWER
I20260924 11:39:17.101402 27938 consensus_queue.cc:260] T 00000000000000000000000000000000 P d86f01415720495abfee7c5d7730d553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d86f01415720495abfee7c5d7730d553" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43627 } }
I20260924 11:39:17.101641 27938 raft_consensus.cc:388] T 00000000000000000000000000000000 P d86f01415720495abfee7c5d7730d553 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:39:17.101797 27938 raft_consensus.cc:482] T 00000000000000000000000000000000 P d86f01415720495abfee7c5d7730d553 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:39:17.101985 27938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d86f01415720495abfee7c5d7730d553 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:39:17.104375 27938 raft_consensus.cc:504] T 00000000000000000000000000000000 P d86f01415720495abfee7c5d7730d553 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d86f01415720495abfee7c5d7730d553" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43627 } }
I20260924 11:39:17.104947 27938 leader_election.cc:302] T 00000000000000000000000000000000 P d86f01415720495abfee7c5d7730d553 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d86f01415720495abfee7c5d7730d553; no voters:
I20260924 11:39:17.105643 27938 leader_election.cc:288] T 00000000000000000000000000000000 P d86f01415720495abfee7c5d7730d553 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:39:17.105808 27948 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d86f01415720495abfee7c5d7730d553 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:39:17.106619 27948 raft_consensus.cc:686] T 00000000000000000000000000000000 P d86f01415720495abfee7c5d7730d553 [term 1 LEADER]: Becoming Leader. State: Replica: d86f01415720495abfee7c5d7730d553, State: Running, Role: LEADER
I20260924 11:39:17.107267 27948 consensus_queue.cc:237] T 00000000000000000000000000000000 P d86f01415720495abfee7c5d7730d553 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d86f01415720495abfee7c5d7730d553" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43627 } }
I20260924 11:39:17.107843 27938 sys_catalog.cc:564] T 00000000000000000000000000000000 P d86f01415720495abfee7c5d7730d553 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:39:17.110512 27949 sys_catalog.cc:455] T 00000000000000000000000000000000 P d86f01415720495abfee7c5d7730d553 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d86f01415720495abfee7c5d7730d553" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d86f01415720495abfee7c5d7730d553" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43627 } } }
I20260924 11:39:17.110965 27949 sys_catalog.cc:458] T 00000000000000000000000000000000 P d86f01415720495abfee7c5d7730d553 [sys.catalog]: This master's current role is: LEADER
I20260924 11:39:17.111325 27950 sys_catalog.cc:455] T 00000000000000000000000000000000 P d86f01415720495abfee7c5d7730d553 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d86f01415720495abfee7c5d7730d553. Latest consensus state: current_term: 1 leader_uuid: "d86f01415720495abfee7c5d7730d553" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d86f01415720495abfee7c5d7730d553" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43627 } } }
I20260924 11:39:17.111714 27950 sys_catalog.cc:458] T 00000000000000000000000000000000 P d86f01415720495abfee7c5d7730d553 [sys.catalog]: This master's current role is: LEADER
I20260924 11:39:17.111945 27957 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:39:17.122048 27957 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:39:17.131825 27957 catalog_manager.cc:1347] Generated new cluster ID: 7dbb2f8e34ef4b73b44439f08c1557ee
I20260924 11:39:17.131963 27957 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:39:17.158437 27957 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:39:17.159994 27957 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:39:17.174263 27957 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d86f01415720495abfee7c5d7730d553: Generated new TSK 0
I20260924 11:39:17.175244 27957 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:39:17.190675 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 27871
W20260924 11:39:17.191967 27933 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37931
W20260924 11:39:17.192754 6992 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 27871: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:39:17.192899 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 27871
2026-09-24T11:39:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1990 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790249617057452-6992-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:39:17 dist-test-slave-cq7c krb5kdc[27969](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:39:17 dist-test-slave-cq7c krb5kdc[27969](info): set up 2 sockets
Sep 24 11:39:17 dist-test-slave-cq7c krb5kdc[27969](info): commencing operation
krb5kdc: starting...
W20260924 11:39:18.335942 6992 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.002s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Sep 24 11:39:18 dist-test-slave-cq7c krb5kdc[27969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:39:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:39:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790249617057452-6992-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790249617057452-6992-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 11:39:18.627240 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33283
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790249617057452-6992-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:39:18.973596 27982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:39:18.973897 27982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:39:18.974009 27982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:39:18.981608 27982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:39:18.981753 27982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:39:18.981804 27982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:39:18.981866 27982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:39:18.981911 27982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:39:18.991957 27982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790249617057452-6992-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:39:18.993259 27982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:39:18.994675 27982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:39:19.006887 27987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:19.006927 27990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:19.006899 27988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:39:19.007755 27982 server_base.cc:1034] running on GCE node
I20260924 11:39:19.008589 27982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:39:19.010107 27982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:39:19.011436 27982 hybrid_clock.cc:648] HybridClock initialized: now 1790249959011310 us; error 56 us; skew 500 ppm
I20260924 11:39:19.011894 27982 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:39:19 dist-test-slave-cq7c krb5kdc[27969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:39:19.021423 27982 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 11:39:19.023228 27982 webserver.cc:460] Webserver started at http://127.0.0.1:40435/ using document root <none> and password file <none>
I20260924 11:39:19.024056 27982 fs_manager.cc:359] Metadata directory not provided
I20260924 11:39:19.024214 27982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:39:19.024535 27982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:39:19.027725 27982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "4578b383f1b24853acfb0c23e46feddb"
format_stamp: "Formatted at 2026-09-24 11:39:19 on dist-test-slave-cq7c"
server_key: "9d888b1077ba5844eefd630774e614b3"
server_key_iv: "5c90ed18d224197f9003aaa3114304d1"
server_key_version: "encryptionkey@0"
I20260924 11:39:19.028514 27982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "4578b383f1b24853acfb0c23e46feddb"
format_stamp: "Formatted at 2026-09-24 11:39:19 on dist-test-slave-cq7c"
server_key: "9d888b1077ba5844eefd630774e614b3"
server_key_iv: "5c90ed18d224197f9003aaa3114304d1"
server_key_version: "encryptionkey@0"
I20260924 11:39:19.034044 27982 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 11:39:19.037997 27997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:39:19.039520 27982 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260924 11:39:19.039891 27982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "4578b383f1b24853acfb0c23e46feddb"
format_stamp: "Formatted at 2026-09-24 11:39:19 on dist-test-slave-cq7c"
server_key: "9d888b1077ba5844eefd630774e614b3"
server_key_iv: "5c90ed18d224197f9003aaa3114304d1"
server_key_version: "encryptionkey@0"
I20260924 11:39:19.040140 27982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:39:19.046279 27982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:39:19.050819 27982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:39:19.051126 27982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:39:19.071614 27982 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33283
I20260924 11:39:19.071652 28048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33283 every 8 connection(s)
I20260924 11:39:19.073055 27982 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:39:19.075223 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 27982
I20260924 11:39:19.075484 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:39:19.076056 6992 external_mini_cluster.cc:1428] Setting key b7a2a13a5d90726ec4d7492d5ecc3e99
I20260924 11:39:19.077303 28049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:39:19.085119 28052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:19.077999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49895 (local address 127.0.0.1:33283)
0924 11:39:19.078893 (+ 894us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:19.078909 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:19.078963 (+ 54us) server_negotiation.cc:407] Connection header received
0924 11:39:19.080966 (+ 2003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:19.080995 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:19.081498 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:19.081962 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:19.082991 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:19.082999 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:19.083017 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:39:19.083032 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:19.083125 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:39:19.083151 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:19.083407 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:19.083554 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 11:39:19.083751 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":370,"threads_started":1}
I20260924 11:39:19.089074 6992 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:39:19.091974 28049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4578b383f1b24853acfb0c23e46feddb: Bootstrap starting.
I20260924 11:39:19.095721 28052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:19.093366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60493 (local address 127.0.0.1:33283)
0924 11:39:19.093580 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:19.093587 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:19.093812 (+ 225us) server_negotiation.cc:407] Connection header received
0924 11:39:19.093938 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:19.093945 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:19.094009 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:19.094145 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:19.095021 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:19.095027 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:19.095031 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:39:19.095047 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:19.095104 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:39:19.095121 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:19.095239 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:19.095309 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 11:39:19.095390 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 11:39:19.095760 28049 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4578b383f1b24853acfb0c23e46feddb: Neither blocks nor log segments found. Creating new log.
I20260924 11:39:19.096897 28049 log.cc:824] T 00000000000000000000000000000000 P 4578b383f1b24853acfb0c23e46feddb: Log is configured to *not* fsync() on all Append() calls
I20260924 11:39:19.099756 28049 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4578b383f1b24853acfb0c23e46feddb: No bootstrap required, opened a new log
I20260924 11:39:19.106877 28049 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4578b383f1b24853acfb0c23e46feddb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4578b383f1b24853acfb0c23e46feddb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33283 } }
I20260924 11:39:19.107295 28049 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4578b383f1b24853acfb0c23e46feddb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:39:19.107425 28049 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4578b383f1b24853acfb0c23e46feddb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4578b383f1b24853acfb0c23e46feddb, State: Initialized, Role: FOLLOWER
I20260924 11:39:19.108012 28049 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4578b383f1b24853acfb0c23e46feddb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4578b383f1b24853acfb0c23e46feddb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33283 } }
I20260924 11:39:19.108263 28049 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4578b383f1b24853acfb0c23e46feddb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:39:19.108412 28049 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4578b383f1b24853acfb0c23e46feddb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:39:19.108582 28049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4578b383f1b24853acfb0c23e46feddb [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:39:19.110828 28049 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4578b383f1b24853acfb0c23e46feddb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4578b383f1b24853acfb0c23e46feddb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33283 } }
I20260924 11:39:19.111321 28049 leader_election.cc:302] T 00000000000000000000000000000000 P 4578b383f1b24853acfb0c23e46feddb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4578b383f1b24853acfb0c23e46feddb; no voters:
I20260924 11:39:19.112004 28049 leader_election.cc:288] T 00000000000000000000000000000000 P 4578b383f1b24853acfb0c23e46feddb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:39:19.112181 28059 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4578b383f1b24853acfb0c23e46feddb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:39:19.112893 28059 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4578b383f1b24853acfb0c23e46feddb [term 1 LEADER]: Becoming Leader. State: Replica: 4578b383f1b24853acfb0c23e46feddb, State: Running, Role: LEADER
I20260924 11:39:19.113584 28059 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4578b383f1b24853acfb0c23e46feddb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4578b383f1b24853acfb0c23e46feddb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33283 } }
I20260924 11:39:19.114352 28049 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4578b383f1b24853acfb0c23e46feddb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:39:19.117033 28061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4578b383f1b24853acfb0c23e46feddb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4578b383f1b24853acfb0c23e46feddb. Latest consensus state: current_term: 1 leader_uuid: "4578b383f1b24853acfb0c23e46feddb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4578b383f1b24853acfb0c23e46feddb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33283 } } }
I20260924 11:39:19.117468 28061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4578b383f1b24853acfb0c23e46feddb [sys.catalog]: This master's current role is: LEADER
I20260924 11:39:19.117084 28060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4578b383f1b24853acfb0c23e46feddb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4578b383f1b24853acfb0c23e46feddb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4578b383f1b24853acfb0c23e46feddb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33283 } } }
I20260924 11:39:19.117951 28060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4578b383f1b24853acfb0c23e46feddb [sys.catalog]: This master's current role is: LEADER
I20260924 11:39:19.119131 28068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:39:19.124636 28068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:39:19.134220 28068 catalog_manager.cc:1347] Generated new cluster ID: f6c4d903b42e4b51b6ac4097a3a32c03
I20260924 11:39:19.134346 28068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:39:19.169647 28068 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:39:19.170130 28068 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:39:19.180817 28068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4578b383f1b24853acfb0c23e46feddb: Generated new TSK 0
I20260924 11:39:19.181748 28068 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:39:19.191980 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 27982
W20260924 11:39:19.193073 28044 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49895
W20260924 11:39:19.193868 6992 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 27982: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:39:19.194023 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 27982
2026-09-24T11:39:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2002 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790249617057452-6992-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:39:19 dist-test-slave-cq7c krb5kdc[28080](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:39:19 dist-test-slave-cq7c krb5kdc[28080](info): set up 2 sockets
Sep 24 11:39:19 dist-test-slave-cq7c krb5kdc[28080](info): commencing operation
krb5kdc: starting...
W20260924 11:39:20.311937 6992 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Sep 24 11:39:20 dist-test-slave-cq7c krb5kdc[28080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:39:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:39:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790249617057452-6992-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790249617057452-6992-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 11:39:20.602615 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34683
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790249617057452-6992-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:39:20.946712 28093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:39:20.947021 28093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:39:20.947113 28093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:39:20.954658 28093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:39:20.954795 28093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:39:20.954843 28093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:39:20.954903 28093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:39:20.954949 28093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:39:20.965071 28093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790249617057452-6992-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:39:20.966362 28093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:39:20.967689 28093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:39:20.979693 28098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:20.979722 28099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:20.979790 28101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:39:20.980902 28093 server_base.cc:1034] running on GCE node
I20260924 11:39:20.981727 28093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:39:20.983264 28093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:39:20.984501 28093 hybrid_clock.cc:648] HybridClock initialized: now 1790249960984458 us; error 40 us; skew 500 ppm
I20260924 11:39:20.984877 28093 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:39:20 dist-test-slave-cq7c krb5kdc[28080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249960, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:39:20.988162 28093 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 11:39:20.989764 28093 webserver.cc:460] Webserver started at http://127.0.0.1:45505/ using document root <none> and password file <none>
I20260924 11:39:20.990502 28093 fs_manager.cc:359] Metadata directory not provided
I20260924 11:39:20.990648 28093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:39:20.990957 28093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:39:20.994505 28093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "5e3f73b33c41421fb52c0d0d2a472b41"
format_stamp: "Formatted at 2026-09-24 11:39:20 on dist-test-slave-cq7c"
server_key: "0fe8e1fe283164ee67ed4e78cab0a1b3"
server_key_iv: "d4e35ad99fb3d3636957083fc1beed1a"
server_key_version: "encryptionkey@0"
I20260924 11:39:20.995285 28093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "5e3f73b33c41421fb52c0d0d2a472b41"
format_stamp: "Formatted at 2026-09-24 11:39:20 on dist-test-slave-cq7c"
server_key: "0fe8e1fe283164ee67ed4e78cab0a1b3"
server_key_iv: "d4e35ad99fb3d3636957083fc1beed1a"
server_key_version: "encryptionkey@0"
I20260924 11:39:21.000887 28093 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 11:39:21.004621 28108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:39:21.005995 28093 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 11:39:21.006191 28093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "5e3f73b33c41421fb52c0d0d2a472b41"
format_stamp: "Formatted at 2026-09-24 11:39:20 on dist-test-slave-cq7c"
server_key: "0fe8e1fe283164ee67ed4e78cab0a1b3"
server_key_iv: "d4e35ad99fb3d3636957083fc1beed1a"
server_key_version: "encryptionkey@0"
I20260924 11:39:21.006405 28093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:39:21.026535 28093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:39:21.030948 28093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:39:21.031240 28093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:39:21.050396 28093 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34683
I20260924 11:39:21.050426 28159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34683 every 8 connection(s)
I20260924 11:39:21.051928 28093 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:39:21.056211 28160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:39:21.060705 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 28093
I20260924 11:39:21.060892 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:39:21.061456 6992 external_mini_cluster.cc:1428] Setting key 25c2cbd4021b4ec44dc76452e09a8b99
I20260924 11:39:21.067623 28160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e3f73b33c41421fb52c0d0d2a472b41: Bootstrap starting.
I20260924 11:39:21.070720 28160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e3f73b33c41421fb52c0d0d2a472b41: Neither blocks nor log segments found. Creating new log.
I20260924 11:39:21.071851 28160 log.cc:824] T 00000000000000000000000000000000 P 5e3f73b33c41421fb52c0d0d2a472b41: Log is configured to *not* fsync() on all Append() calls
I20260924 11:39:21.075695 28160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e3f73b33c41421fb52c0d0d2a472b41: No bootstrap required, opened a new log
Sep 24 11:39:21 dist-test-slave-cq7c krb5kdc[28080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 11:39:21.090184 28163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:21.063436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.95:34293 (local address 127.0.0.1:34683)
0924 11:39:21.064174 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:21.064186 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:21.064227 (+ 41us) server_negotiation.cc:407] Connection header received
0924 11:39:21.066383 (+ 2156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:21.066406 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:21.066813 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:21.067211 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:21.068684 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:21.070038 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:21.071386 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:21.072226 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:21.072435 (+ 209us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:21.078335 (+ 5900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:21.078349 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:21.078372 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:39:21.078431 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:21.083302 (+ 4871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:21.084947 (+ 1645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:21.084978 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:21.084991 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:21.085298 (+ 307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:21.085829 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:21.085835 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:21.085838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:21.086806 (+ 968us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:21.087206 (+ 400us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:21.087873 (+ 667us) server_negotiation.cc:299] Negotiation successful
0924 11:39:21.088266 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":310,"threads_started":1}
I20260924 11:39:21.091588 28160 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e3f73b33c41421fb52c0d0d2a472b41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e3f73b33c41421fb52c0d0d2a472b41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34683 } }
I20260924 11:39:21.092137 28160 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e3f73b33c41421fb52c0d0d2a472b41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:39:21.092420 28160 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e3f73b33c41421fb52c0d0d2a472b41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e3f73b33c41421fb52c0d0d2a472b41, State: Initialized, Role: FOLLOWER
I20260924 11:39:21.093231 28160 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e3f73b33c41421fb52c0d0d2a472b41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e3f73b33c41421fb52c0d0d2a472b41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34683 } }
I20260924 11:39:21.093545 28160 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e3f73b33c41421fb52c0d0d2a472b41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:39:21.093786 28160 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e3f73b33c41421fb52c0d0d2a472b41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:39:21.093999 28160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e3f73b33c41421fb52c0d0d2a472b41 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:39:21.096621 6992 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:39:21.097592 28160 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e3f73b33c41421fb52c0d0d2a472b41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e3f73b33c41421fb52c0d0d2a472b41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34683 } }
I20260924 11:39:21.098361 28160 leader_election.cc:302] T 00000000000000000000000000000000 P 5e3f73b33c41421fb52c0d0d2a472b41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e3f73b33c41421fb52c0d0d2a472b41; no voters:
I20260924 11:39:21.099139 28160 leader_election.cc:288] T 00000000000000000000000000000000 P 5e3f73b33c41421fb52c0d0d2a472b41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:39:21.099282 28165 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e3f73b33c41421fb52c0d0d2a472b41 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:39:21.101696 28160 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e3f73b33c41421fb52c0d0d2a472b41 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:39:21.106559 28165 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e3f73b33c41421fb52c0d0d2a472b41 [term 1 LEADER]: Becoming Leader. State: Replica: 5e3f73b33c41421fb52c0d0d2a472b41, State: Running, Role: LEADER
I20260924 11:39:21.107481 28165 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e3f73b33c41421fb52c0d0d2a472b41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e3f73b33c41421fb52c0d0d2a472b41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34683 } }
I20260924 11:39:21.111606 28166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e3f73b33c41421fb52c0d0d2a472b41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e3f73b33c41421fb52c0d0d2a472b41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e3f73b33c41421fb52c0d0d2a472b41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34683 } } }
I20260924 11:39:21.112115 28166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e3f73b33c41421fb52c0d0d2a472b41 [sys.catalog]: This master's current role is: LEADER
I20260924 11:39:21.114580 28167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e3f73b33c41421fb52c0d0d2a472b41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e3f73b33c41421fb52c0d0d2a472b41. Latest consensus state: current_term: 1 leader_uuid: "5e3f73b33c41421fb52c0d0d2a472b41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e3f73b33c41421fb52c0d0d2a472b41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34683 } } }
I20260924 11:39:21.114914 28167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e3f73b33c41421fb52c0d0d2a472b41 [sys.catalog]: This master's current role is: LEADER
I20260924 11:39:21.115252 28174 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:39:21.125135 28174 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:39:21.144071 28174 catalog_manager.cc:1347] Generated new cluster ID: de5cce5c50484570960264d95a7adaf3
I20260924 11:39:21.144210 28174 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:39:21.158037 28163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:21.122462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.95:54659 (local address 127.0.0.1:34683)
0924 11:39:21.122686 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:21.122692 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:21.139528 (+ 16836us) server_negotiation.cc:407] Connection header received
0924 11:39:21.139630 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:21.139638 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:21.139766 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:21.139941 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:21.142740 (+ 2799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:21.144317 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:21.146966 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:21.147894 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:21.148035 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:21.151222 (+ 3187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:21.151228 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:21.151231 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:39:21.151267 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:21.154642 (+ 3375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:21.155570 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:21.155575 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:21.155578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:21.155721 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:21.156258 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:21.156265 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:21.156274 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:21.156610 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:21.156916 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:21.157361 (+ 445us) server_negotiation.cc:299] Negotiation successful
0924 11:39:21.157598 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 11:39:21.170722 28174 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:39:21.172695 28174 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:39:21.184000 28174 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e3f73b33c41421fb52c0d0d2a472b41: Generated new TSK 0
I20260924 11:39:21.184947 28174 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:39:21.197014 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 28093
I20260924 11:39:21.198246 28155 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:39:21.295781 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 28093
W20260924 11:39:21.314651 28078 connection.cc:535] client connection to 127.0.0.1:34683 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34683): Connection reset by peer (error 104)
2026-09-24T11:39:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2102 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790249617057452-6992-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:39:21 dist-test-slave-cq7c krb5kdc[28193](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:39:21 dist-test-slave-cq7c krb5kdc[28193](info): set up 2 sockets
Sep 24 11:39:21 dist-test-slave-cq7c krb5kdc[28193](info): commencing operation
krb5kdc: starting...
W20260924 11:39:22.435914 6992 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.004s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Sep 24 11:39:22 dist-test-slave-cq7c krb5kdc[28193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249962, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:39:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:39:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790249617057452-6992-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790249617057452-6992-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 11:39:22.732208 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45459
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790249617057452-6992-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:39:23.076478 28206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:39:23.076788 28206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:39:23.076905 28206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:39:23.084399 28206 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:39:23.084520 28206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:39:23.084568 28206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:39:23.084622 28206 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:39:23.084658 28206 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:39:23.094707 28206 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790249617057452-6992-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:39:23.095984 28206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:39:23.097235 28206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:39:23.109323 28214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:23.109293 28212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:23.109663 28211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:39:23.110215 28206 server_base.cc:1034] running on GCE node
I20260924 11:39:23.111054 28206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:39:23.112669 28206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:39:23.113912 28206 hybrid_clock.cc:648] HybridClock initialized: now 1790249963113868 us; error 39 us; skew 500 ppm
I20260924 11:39:23.114319 28206 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:39:23 dist-test-slave-cq7c krb5kdc[28193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:39:23.117631 28206 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 11:39:23.119207 28206 webserver.cc:460] Webserver started at http://127.0.0.1:40397/ using document root <none> and password file <none>
I20260924 11:39:23.120041 28206 fs_manager.cc:359] Metadata directory not provided
I20260924 11:39:23.120198 28206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:39:23.120534 28206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:39:23.123939 28206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "d3d054fe9c994cc4a40635811047b409"
format_stamp: "Formatted at 2026-09-24 11:39:23 on dist-test-slave-cq7c"
server_key: "f2b0fe09ef24448b8c30bf3e291c30b6"
server_key_iv: "35da8307652baed00144966c92fc6512"
server_key_version: "encryptionkey@0"
I20260924 11:39:23.124831 28206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "d3d054fe9c994cc4a40635811047b409"
format_stamp: "Formatted at 2026-09-24 11:39:23 on dist-test-slave-cq7c"
server_key: "f2b0fe09ef24448b8c30bf3e291c30b6"
server_key_iv: "35da8307652baed00144966c92fc6512"
server_key_version: "encryptionkey@0"
I20260924 11:39:23.130491 28206 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 11:39:23.134362 28221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:39:23.135905 28206 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:39:23.136102 28206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "d3d054fe9c994cc4a40635811047b409"
format_stamp: "Formatted at 2026-09-24 11:39:23 on dist-test-slave-cq7c"
server_key: "f2b0fe09ef24448b8c30bf3e291c30b6"
server_key_iv: "35da8307652baed00144966c92fc6512"
server_key_version: "encryptionkey@0"
I20260924 11:39:23.136307 28206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:39:23.153293 28206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:39:23.157950 28206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:39:23.158281 28206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:39:23.177932 28206 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45459
I20260924 11:39:23.177956 28272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45459 every 8 connection(s)
I20260924 11:39:23.179425 28206 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:39:23.180982 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 28206
I20260924 11:39:23.181160 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:39:23.181792 6992 external_mini_cluster.cc:1428] Setting key d89ad423c50e6ea1a61a951403361a9c
I20260924 11:39:23.184262 28273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:39:23.196158 28276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:23.183712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.95:43707 (local address 127.0.0.1:45459)
0924 11:39:23.184534 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:23.184547 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:23.184593 (+ 46us) server_negotiation.cc:407] Connection header received
0924 11:39:23.186497 (+ 1904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:23.186528 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:23.187157 (+ 629us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:23.187595 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:23.188218 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:23.189586 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:23.191497 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:23.192229 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:23.192362 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:23.193687 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:23.193698 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:23.193713 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:39:23.194305 (+ 592us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:23.194394 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:39:23.194423 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:23.194551 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:23.194837 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 11:39:23.195070 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":364,"threads_started":1}
I20260924 11:39:23.197515 28273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3d054fe9c994cc4a40635811047b409: Bootstrap starting.
I20260924 11:39:23.200237 6992 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:39:23.200946 28273 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3d054fe9c994cc4a40635811047b409: Neither blocks nor log segments found. Creating new log.
I20260924 11:39:23.202204 28273 log.cc:824] T 00000000000000000000000000000000 P d3d054fe9c994cc4a40635811047b409: Log is configured to *not* fsync() on all Append() calls
I20260924 11:39:23.205668 28273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3d054fe9c994cc4a40635811047b409: No bootstrap required, opened a new log
I20260924 11:39:23.211031 28276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:23.204825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.95:60587 (local address 127.0.0.1:45459)
0924 11:39:23.205034 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:23.205041 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:23.205406 (+ 365us) server_negotiation.cc:407] Connection header received
0924 11:39:23.205587 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:23.205591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:23.205665 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:23.205781 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:23.206285 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:23.207036 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:23.208427 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:23.208900 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:23.209056 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:23.210265 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:23.210269 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:23.210272 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:39:23.210290 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:23.210328 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:39:23.210343 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:23.210474 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:23.210663 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 11:39:23.210729 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 11:39:23.214092 28273 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3d054fe9c994cc4a40635811047b409 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d054fe9c994cc4a40635811047b409" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45459 } }
I20260924 11:39:23.214488 28273 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3d054fe9c994cc4a40635811047b409 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:39:23.214603 28273 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3d054fe9c994cc4a40635811047b409 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3d054fe9c994cc4a40635811047b409, State: Initialized, Role: FOLLOWER
I20260924 11:39:23.215176 28273 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3d054fe9c994cc4a40635811047b409 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d054fe9c994cc4a40635811047b409" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45459 } }
I20260924 11:39:23.215453 28273 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3d054fe9c994cc4a40635811047b409 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:39:23.215610 28273 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3d054fe9c994cc4a40635811047b409 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:39:23.215795 28273 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3d054fe9c994cc4a40635811047b409 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:39:23.218223 28273 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3d054fe9c994cc4a40635811047b409 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d054fe9c994cc4a40635811047b409" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45459 } }
I20260924 11:39:23.218878 28273 leader_election.cc:302] T 00000000000000000000000000000000 P d3d054fe9c994cc4a40635811047b409 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3d054fe9c994cc4a40635811047b409; no voters:
I20260924 11:39:23.219628 28273 leader_election.cc:288] T 00000000000000000000000000000000 P d3d054fe9c994cc4a40635811047b409 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:39:23.219766 28283 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3d054fe9c994cc4a40635811047b409 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:39:23.220504 28283 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3d054fe9c994cc4a40635811047b409 [term 1 LEADER]: Becoming Leader. State: Replica: d3d054fe9c994cc4a40635811047b409, State: Running, Role: LEADER
I20260924 11:39:23.221047 28283 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3d054fe9c994cc4a40635811047b409 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d054fe9c994cc4a40635811047b409" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45459 } }
I20260924 11:39:23.221858 28273 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3d054fe9c994cc4a40635811047b409 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:39:23.223989 28285 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3d054fe9c994cc4a40635811047b409 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3d054fe9c994cc4a40635811047b409. Latest consensus state: current_term: 1 leader_uuid: "d3d054fe9c994cc4a40635811047b409" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d054fe9c994cc4a40635811047b409" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45459 } } }
I20260924 11:39:23.224418 28285 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3d054fe9c994cc4a40635811047b409 [sys.catalog]: This master's current role is: LEADER
I20260924 11:39:23.224289 28284 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3d054fe9c994cc4a40635811047b409 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3d054fe9c994cc4a40635811047b409" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3d054fe9c994cc4a40635811047b409" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45459 } } }
I20260924 11:39:23.224588 28284 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3d054fe9c994cc4a40635811047b409 [sys.catalog]: This master's current role is: LEADER
I20260924 11:39:23.226394 28290 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:39:23.232278 28290 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:39:23.241657 28290 catalog_manager.cc:1347] Generated new cluster ID: f13e54b62afd41c2a6032a732ae92047
I20260924 11:39:23.241791 28290 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:39:23.261804 28290 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:39:23.263126 28290 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:39:23.271476 28290 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3d054fe9c994cc4a40635811047b409: Generated new TSK 0
I20260924 11:39:23.272344 28290 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:39:23.281140 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 28206
W20260924 11:39:23.282364 28268 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.95:43707
W20260924 11:39:23.283133 6992 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 28206: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:39:23.283257 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 28206
W20260924 11:39:23.301074 28191 connection.cc:535] client connection to 127.0.0.1:45459 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45459): Connection reset by peer (error 104)
2026-09-24T11:39:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1986 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790249617057452-6992-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:39:23 dist-test-slave-cq7c krb5kdc[28304](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:39:23 dist-test-slave-cq7c krb5kdc[28304](info): set up 2 sockets
Sep 24 11:39:23 dist-test-slave-cq7c krb5kdc[28304](info): commencing operation
krb5kdc: starting...
W20260924 11:39:24.423811 6992 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Sep 24 11:39:24 dist-test-slave-cq7c krb5kdc[28304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:39:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:39:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790249617057452-6992-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790249617057452-6992-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 11:39:24.699123 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36183
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790249617057452-6992-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36183
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:39:25.044610 28317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:39:25.044970 28317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:39:25.045130 28317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:39:25.052394 28317 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:39:25.052563 28317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:39:25.052675 28317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:39:25.052791 28317 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:39:25.052870 28317 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:39:25.063550 28317 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790249617057452-6992-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:39:25.064924 28317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:39:25.066319 28317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:39:25.079035 28322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:25.079083 28323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:25.078994 28325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:39:25.080406 28317 server_base.cc:1034] running on GCE node
I20260924 11:39:25.081280 28317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:39:25.082894 28317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:39:25.084126 28317 hybrid_clock.cc:648] HybridClock initialized: now 1790249965084078 us; error 44 us; skew 500 ppm
I20260924 11:39:25.084496 28317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:39:25 dist-test-slave-cq7c krb5kdc[28304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249965, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:39:25.094022 28317 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 11:39:25.095777 28317 webserver.cc:460] Webserver started at http://127.0.0.1:39965/ using document root <none> and password file <none>
I20260924 11:39:25.096504 28317 fs_manager.cc:359] Metadata directory not provided
I20260924 11:39:25.096657 28317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:39:25.096966 28317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:39:25.100102 28317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "afcc21db647945cea4f57be0b5bb3cd7"
format_stamp: "Formatted at 2026-09-24 11:39:25 on dist-test-slave-cq7c"
server_key: "e1fb7e58296829caf38f015aa745ae5e"
server_key_iv: "b44144443d4bd8ce2593bc399515d3fd"
server_key_version: "encryptionkey@0"
I20260924 11:39:25.100845 28317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "afcc21db647945cea4f57be0b5bb3cd7"
format_stamp: "Formatted at 2026-09-24 11:39:25 on dist-test-slave-cq7c"
server_key: "e1fb7e58296829caf38f015aa745ae5e"
server_key_iv: "b44144443d4bd8ce2593bc399515d3fd"
server_key_version: "encryptionkey@0"
I20260924 11:39:25.106065 28317 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 11:39:25.109721 28332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:39:25.111083 28317 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 11:39:25.111287 28317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "afcc21db647945cea4f57be0b5bb3cd7"
format_stamp: "Formatted at 2026-09-24 11:39:25 on dist-test-slave-cq7c"
server_key: "e1fb7e58296829caf38f015aa745ae5e"
server_key_iv: "b44144443d4bd8ce2593bc399515d3fd"
server_key_version: "encryptionkey@0"
I20260924 11:39:25.111565 28317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:39:25.118052 28317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:39:25.122210 28317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:39:25.122457 28317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:39:25.141131 28317 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36183
I20260924 11:39:25.141173 28383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36183 every 8 connection(s)
I20260924 11:39:25.142706 28317 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:39:25.146714 28384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:39:25.148702 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 28317
I20260924 11:39:25.148891 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:39:25.149351 6992 external_mini_cluster.cc:1428] Setting key cbd15472034203e0d9a52b708d6f8474
I20260924 11:39:25.158213 28387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:25.151292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.95:51361 (local address 127.0.0.1:36183)
0924 11:39:25.152247 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:25.152260 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:25.152311 (+ 51us) server_negotiation.cc:407] Connection header received
0924 11:39:25.154370 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:25.154395 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:25.154793 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:25.155261 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:25.156474 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:25.156484 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:25.156497 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:39:25.156512 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:25.156579 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:39:25.156600 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:25.156880 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:25.157031 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 11:39:25.157223 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":423,"threads_started":1}
I20260924 11:39:25.159068 28384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afcc21db647945cea4f57be0b5bb3cd7: Bootstrap starting.
I20260924 11:39:25.162413 6992 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:39:25.162704 28384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P afcc21db647945cea4f57be0b5bb3cd7: Neither blocks nor log segments found. Creating new log.
I20260924 11:39:25.163807 28384 log.cc:824] T 00000000000000000000000000000000 P afcc21db647945cea4f57be0b5bb3cd7: Log is configured to *not* fsync() on all Append() calls
I20260924 11:39:25.167312 28384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afcc21db647945cea4f57be0b5bb3cd7: No bootstrap required, opened a new log
I20260924 11:39:25.169615 28387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:25.166968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.95:40243 (local address 127.0.0.1:36183)
0924 11:39:25.167130 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:25.167136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:25.167606 (+ 470us) server_negotiation.cc:407] Connection header received
0924 11:39:25.167754 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:25.167762 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:25.167841 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:25.167967 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:25.168926 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:25.168932 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:25.168937 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:39:25.168954 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:25.169007 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:39:25.169031 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:25.169162 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:25.169229 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 11:39:25.169306 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 11:39:25.176429 28384 raft_consensus.cc:348] T 00000000000000000000000000000000 P afcc21db647945cea4f57be0b5bb3cd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afcc21db647945cea4f57be0b5bb3cd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36183 } }
I20260924 11:39:25.176841 28384 raft_consensus.cc:374] T 00000000000000000000000000000000 P afcc21db647945cea4f57be0b5bb3cd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:39:25.176916 28384 raft_consensus.cc:729] T 00000000000000000000000000000000 P afcc21db647945cea4f57be0b5bb3cd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afcc21db647945cea4f57be0b5bb3cd7, State: Initialized, Role: FOLLOWER
I20260924 11:39:25.177467 28384 consensus_queue.cc:260] T 00000000000000000000000000000000 P afcc21db647945cea4f57be0b5bb3cd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afcc21db647945cea4f57be0b5bb3cd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36183 } }
I20260924 11:39:25.177699 28384 raft_consensus.cc:388] T 00000000000000000000000000000000 P afcc21db647945cea4f57be0b5bb3cd7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:39:25.177845 28384 raft_consensus.cc:482] T 00000000000000000000000000000000 P afcc21db647945cea4f57be0b5bb3cd7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:39:25.177995 28384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afcc21db647945cea4f57be0b5bb3cd7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:39:25.180774 28384 raft_consensus.cc:504] T 00000000000000000000000000000000 P afcc21db647945cea4f57be0b5bb3cd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afcc21db647945cea4f57be0b5bb3cd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36183 } }
I20260924 11:39:25.181245 28384 leader_election.cc:302] T 00000000000000000000000000000000 P afcc21db647945cea4f57be0b5bb3cd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: afcc21db647945cea4f57be0b5bb3cd7; no voters:
I20260924 11:39:25.181802 28384 leader_election.cc:288] T 00000000000000000000000000000000 P afcc21db647945cea4f57be0b5bb3cd7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:39:25.181962 28394 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afcc21db647945cea4f57be0b5bb3cd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:39:25.182673 28394 raft_consensus.cc:686] T 00000000000000000000000000000000 P afcc21db647945cea4f57be0b5bb3cd7 [term 1 LEADER]: Becoming Leader. State: Replica: afcc21db647945cea4f57be0b5bb3cd7, State: Running, Role: LEADER
I20260924 11:39:25.183272 28394 consensus_queue.cc:237] T 00000000000000000000000000000000 P afcc21db647945cea4f57be0b5bb3cd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afcc21db647945cea4f57be0b5bb3cd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36183 } }
I20260924 11:39:25.184130 28384 sys_catalog.cc:564] T 00000000000000000000000000000000 P afcc21db647945cea4f57be0b5bb3cd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:39:25.186050 28395 sys_catalog.cc:455] T 00000000000000000000000000000000 P afcc21db647945cea4f57be0b5bb3cd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "afcc21db647945cea4f57be0b5bb3cd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afcc21db647945cea4f57be0b5bb3cd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36183 } } }
I20260924 11:39:25.186503 28395 sys_catalog.cc:458] T 00000000000000000000000000000000 P afcc21db647945cea4f57be0b5bb3cd7 [sys.catalog]: This master's current role is: LEADER
I20260924 11:39:25.186117 28396 sys_catalog.cc:455] T 00000000000000000000000000000000 P afcc21db647945cea4f57be0b5bb3cd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader afcc21db647945cea4f57be0b5bb3cd7. Latest consensus state: current_term: 1 leader_uuid: "afcc21db647945cea4f57be0b5bb3cd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afcc21db647945cea4f57be0b5bb3cd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36183 } } }
I20260924 11:39:25.187107 28396 sys_catalog.cc:458] T 00000000000000000000000000000000 P afcc21db647945cea4f57be0b5bb3cd7 [sys.catalog]: This master's current role is: LEADER
I20260924 11:39:25.192710 28401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:39:25.198467 28401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:39:25.207836 28401 catalog_manager.cc:1347] Generated new cluster ID: 805a7e0f59134ebda0d97f17770b5a25
I20260924 11:39:25.207964 28401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:39:25.265573 28401 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:39:25.266152 28401 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:39:25.281719 28401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P afcc21db647945cea4f57be0b5bb3cd7: Generated new TSK 0
I20260924 11:39:25.282604 28401 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:39:25.303970 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 28317
W20260924 11:39:25.305276 28379 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.95:51361
W20260924 11:39:25.306162 6992 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 28317: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:39:25.306344 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 28317
2026-09-24T11:39:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2025 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790249617057452-6992-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:39:25 dist-test-slave-cq7c krb5kdc[28415](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:39:25 dist-test-slave-cq7c krb5kdc[28415](info): set up 2 sockets
Sep 24 11:39:25 dist-test-slave-cq7c krb5kdc[28415](info): commencing operation
krb5kdc: starting...
W20260924 11:39:26.427711 6992 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.000s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Sep 24 11:39:26 dist-test-slave-cq7c krb5kdc[28415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:39:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:39:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790249617057452-6992-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790249617057452-6992-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 11:39:26.711293 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36249
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790249617057452-6992-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:39:27.050141 28428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:39:27.050448 28428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:39:27.050670 28428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:39:27.058163 28428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:39:27.058285 28428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:39:27.058413 28428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:39:27.058545 28428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:39:27.058616 28428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:39:27.068879 28428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36249
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790249617057452-6992-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:39:27.070282 28428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:39:27.071899 28428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:39:27.085090 28433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:27.085146 28434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:27.085044 28436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:39:27.086285 28428 server_base.cc:1034] running on GCE node
I20260924 11:39:27.087175 28428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:39:27.088783 28428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:39:27.090000 28428 hybrid_clock.cc:648] HybridClock initialized: now 1790249967089922 us; error 72 us; skew 500 ppm
I20260924 11:39:27.090342 28428 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:39:27 dist-test-slave-cq7c krb5kdc[28415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:39:27.093497 28428 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 11:39:27.095073 28428 webserver.cc:460] Webserver started at http://127.0.0.1:34363/ using document root <none> and password file <none>
I20260924 11:39:27.095844 28428 fs_manager.cc:359] Metadata directory not provided
I20260924 11:39:27.095988 28428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:39:27.096293 28428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:39:27.099438 28428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "ba5f7943d94544f1beee0ce32a4f1743"
format_stamp: "Formatted at 2026-09-24 11:39:27 on dist-test-slave-cq7c"
server_key: "e7f6160a86c232980f6227894ca14307"
server_key_iv: "892661ac56ee3a7f375934369dd79d57"
server_key_version: "encryptionkey@0"
I20260924 11:39:27.100247 28428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "ba5f7943d94544f1beee0ce32a4f1743"
format_stamp: "Formatted at 2026-09-24 11:39:27 on dist-test-slave-cq7c"
server_key: "e7f6160a86c232980f6227894ca14307"
server_key_iv: "892661ac56ee3a7f375934369dd79d57"
server_key_version: "encryptionkey@0"
I20260924 11:39:27.105423 28428 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 11:39:27.109099 28443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:39:27.110445 28428 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 11:39:27.110710 28428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "ba5f7943d94544f1beee0ce32a4f1743"
format_stamp: "Formatted at 2026-09-24 11:39:27 on dist-test-slave-cq7c"
server_key: "e7f6160a86c232980f6227894ca14307"
server_key_iv: "892661ac56ee3a7f375934369dd79d57"
server_key_version: "encryptionkey@0"
I20260924 11:39:27.110915 28428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:39:27.124728 28428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:39:27.129220 28428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:39:27.129510 28428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:39:27.148914 28428 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36249
I20260924 11:39:27.148952 28494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36249 every 8 connection(s)
I20260924 11:39:27.150367 28428 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:39:27.154414 28495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:39:27.158569 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 28428
I20260924 11:39:27.158747 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:39:27.159169 6992 external_mini_cluster.cc:1428] Setting key cddc3c20ace818b225480da3668b692d
I20260924 11:39:27.168118 28495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba5f7943d94544f1beee0ce32a4f1743: Bootstrap starting.
I20260924 11:39:27.171792 28495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba5f7943d94544f1beee0ce32a4f1743: Neither blocks nor log segments found. Creating new log.
I20260924 11:39:27.172833 28495 log.cc:824] T 00000000000000000000000000000000 P ba5f7943d94544f1beee0ce32a4f1743: Log is configured to *not* fsync() on all Append() calls
Sep 24 11:39:27 dist-test-slave-cq7c krb5kdc[28415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 11:39:27.176105 28495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba5f7943d94544f1beee0ce32a4f1743: No bootstrap required, opened a new log
I20260924 11:39:27.184065 28498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:27.161086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.95:36917 (local address 127.0.0.1:36249)
0924 11:39:27.161840 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:27.161856 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:27.161909 (+ 53us) server_negotiation.cc:407] Connection header received
0924 11:39:27.163743 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:27.163774 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:27.164357 (+ 583us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:27.164811 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:27.166066 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:27.168022 (+ 1956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:27.169400 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:27.170490 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:27.170670 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:27.174769 (+ 4099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:27.174784 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:27.174807 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:39:27.174849 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:27.178890 (+ 4041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:27.179836 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:27.179844 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:27.179855 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:27.180066 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:27.180533 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:27.180538 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:27.180542 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:27.181324 (+ 782us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:27.181688 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:27.182193 (+ 505us) server_negotiation.cc:299] Negotiation successful
0924 11:39:27.182555 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":285,"threads_started":1}
I20260924 11:39:27.187028 28495 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba5f7943d94544f1beee0ce32a4f1743 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba5f7943d94544f1beee0ce32a4f1743" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36249 } }
I20260924 11:39:27.187503 28495 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba5f7943d94544f1beee0ce32a4f1743 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:39:27.187656 28495 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba5f7943d94544f1beee0ce32a4f1743 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba5f7943d94544f1beee0ce32a4f1743, State: Initialized, Role: FOLLOWER
I20260924 11:39:27.188252 28495 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba5f7943d94544f1beee0ce32a4f1743 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba5f7943d94544f1beee0ce32a4f1743" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36249 } }
I20260924 11:39:27.188501 28495 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba5f7943d94544f1beee0ce32a4f1743 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:39:27.188496 6992 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:39:27.188661 28495 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba5f7943d94544f1beee0ce32a4f1743 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:39:27.188833 28495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba5f7943d94544f1beee0ce32a4f1743 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:39:27.190984 28495 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba5f7943d94544f1beee0ce32a4f1743 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba5f7943d94544f1beee0ce32a4f1743" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36249 } }
I20260924 11:39:27.191574 28495 leader_election.cc:302] T 00000000000000000000000000000000 P ba5f7943d94544f1beee0ce32a4f1743 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba5f7943d94544f1beee0ce32a4f1743; no voters:
I20260924 11:39:27.192256 28495 leader_election.cc:288] T 00000000000000000000000000000000 P ba5f7943d94544f1beee0ce32a4f1743 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:39:27.192363 28504 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba5f7943d94544f1beee0ce32a4f1743 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:39:27.194371 28504 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba5f7943d94544f1beee0ce32a4f1743 [term 1 LEADER]: Becoming Leader. State: Replica: ba5f7943d94544f1beee0ce32a4f1743, State: Running, Role: LEADER
I20260924 11:39:27.194787 28495 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba5f7943d94544f1beee0ce32a4f1743 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:39:27.195019 28504 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba5f7943d94544f1beee0ce32a4f1743 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba5f7943d94544f1beee0ce32a4f1743" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36249 } }
I20260924 11:39:27.198191 28505 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba5f7943d94544f1beee0ce32a4f1743 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba5f7943d94544f1beee0ce32a4f1743" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba5f7943d94544f1beee0ce32a4f1743" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36249 } } }
I20260924 11:39:27.198642 28505 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba5f7943d94544f1beee0ce32a4f1743 [sys.catalog]: This master's current role is: LEADER
I20260924 11:39:27.205161 28506 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba5f7943d94544f1beee0ce32a4f1743 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba5f7943d94544f1beee0ce32a4f1743. Latest consensus state: current_term: 1 leader_uuid: "ba5f7943d94544f1beee0ce32a4f1743" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba5f7943d94544f1beee0ce32a4f1743" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36249 } } }
I20260924 11:39:27.205802 28506 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba5f7943d94544f1beee0ce32a4f1743 [sys.catalog]: This master's current role is: LEADER
I20260924 11:39:27.206333 28511 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:39:27.212716 28511 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:39:27.213150 28498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:27.193555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.95:55525 (local address 127.0.0.1:36249)
0924 11:39:27.193714 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:27.193721 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:27.193804 (+ 83us) server_negotiation.cc:407] Connection header received
0924 11:39:27.194018 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:27.194026 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:27.194130 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:27.194214 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:27.195969 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:27.196811 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:27.198635 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:27.199552 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:27.199711 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:27.204318 (+ 4607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:27.204328 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:27.204336 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:39:27.204390 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:27.208862 (+ 4472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:27.209909 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:27.209935 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:27.209939 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:27.210090 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:27.210666 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:27.210671 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:27.210674 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:27.211149 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:27.211442 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:27.212604 (+ 1162us) server_negotiation.cc:299] Negotiation successful
0924 11:39:27.212801 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 11:39:27.223038 28511 catalog_manager.cc:1347] Generated new cluster ID: 9821b20b84224d3494fcd34e48c4b63b
I20260924 11:39:27.223188 28511 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:39:27.231943 28511 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:39:27.233554 28511 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:39:27.241077 28511 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba5f7943d94544f1beee0ce32a4f1743: Generated new TSK 0
I20260924 11:39:27.242004 28511 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:39:27.253099 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 28428
I20260924 11:39:27.254701 28490 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:39:27.347904 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 28428
W20260924 11:39:27.365458 28413 connection.cc:535] client connection to 127.0.0.1:36249 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36249): Connection reset by peer (error 104)
2026-09-24T11:39:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2039 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790249617057452-6992-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:39:27 dist-test-slave-cq7c krb5kdc[28528](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:39:27 dist-test-slave-cq7c krb5kdc[28528](info): set up 2 sockets
Sep 24 11:39:27 dist-test-slave-cq7c krb5kdc[28528](info): commencing operation
krb5kdc: starting...
W20260924 11:39:28.488512 6992 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.001s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Sep 24 11:39:28 dist-test-slave-cq7c krb5kdc[28528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:39:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:39:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790249617057452-6992-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790249617057452-6992-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 11:39:28.770852 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39087
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790249617057452-6992-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39087
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:39:29.106038 28541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:39:29.106424 28541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:39:29.106622 28541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:39:29.113724 28541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:39:29.113881 28541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:39:29.113991 28541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:39:29.114108 28541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:39:29.114197 28541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:39:29.123970 28541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39087
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790249617057452-6992-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:39:29.125347 28541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:39:29.126637 28541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:39:29.138567 28546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:29.138517 28547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:29.139285 28549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:39:29.139683 28541 server_base.cc:1034] running on GCE node
I20260924 11:39:29.140458 28541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:39:29.142000 28541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:39:29.143261 28541 hybrid_clock.cc:648] HybridClock initialized: now 1790249969143204 us; error 53 us; skew 500 ppm
I20260924 11:39:29.143692 28541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:39:29 dist-test-slave-cq7c krb5kdc[28528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249969, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:39:29.147207 28541 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 11:39:29.149003 28541 webserver.cc:460] Webserver started at http://127.0.0.1:46601/ using document root <none> and password file <none>
I20260924 11:39:29.149710 28541 fs_manager.cc:359] Metadata directory not provided
I20260924 11:39:29.149858 28541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:39:29.150179 28541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:39:29.153246 28541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "b3ad0f920fc74e809c79af06b743b567"
format_stamp: "Formatted at 2026-09-24 11:39:29 on dist-test-slave-cq7c"
server_key: "84f639068a739fb858a7b61a9a6bbc4b"
server_key_iv: "eb95222a79fa28cbdc54f21bfefb9bc4"
server_key_version: "encryptionkey@0"
I20260924 11:39:29.154033 28541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "b3ad0f920fc74e809c79af06b743b567"
format_stamp: "Formatted at 2026-09-24 11:39:29 on dist-test-slave-cq7c"
server_key: "84f639068a739fb858a7b61a9a6bbc4b"
server_key_iv: "eb95222a79fa28cbdc54f21bfefb9bc4"
server_key_version: "encryptionkey@0"
I20260924 11:39:29.159410 28541 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 11:39:29.163038 28556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:39:29.164670 28541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 11:39:29.164885 28541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "b3ad0f920fc74e809c79af06b743b567"
format_stamp: "Formatted at 2026-09-24 11:39:29 on dist-test-slave-cq7c"
server_key: "84f639068a739fb858a7b61a9a6bbc4b"
server_key_iv: "eb95222a79fa28cbdc54f21bfefb9bc4"
server_key_version: "encryptionkey@0"
I20260924 11:39:29.165102 28541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:39:29.191706 28541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:39:29.196079 28541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:39:29.196398 28541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:39:29.215186 28541 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39087
I20260924 11:39:29.215231 28607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39087 every 8 connection(s)
I20260924 11:39:29.216642 28541 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:39:29.218204 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 28541
I20260924 11:39:29.218400 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:39:29.219005 6992 external_mini_cluster.cc:1428] Setting key aedc132ca059b592728d9c30b0419661
I20260924 11:39:29.221539 28608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:39:29.232733 28611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:29.220875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.95:45395 (local address 127.0.0.1:39087)
0924 11:39:29.221774 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:29.221791 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:29.221846 (+ 55us) server_negotiation.cc:407] Connection header received
0924 11:39:29.223530 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:29.223556 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:29.223960 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:29.224338 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:29.224892 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:29.226222 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:29.228129 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:29.228811 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:29.228936 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:29.230276 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:29.230283 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:29.230296 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:39:29.230900 (+ 604us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:29.230977 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:39:29.231001 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:29.231126 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:29.231515 (+ 389us) server_negotiation.cc:299] Negotiation successful
0924 11:39:29.231763 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":359,"threads_started":1}
I20260924 11:39:29.234230 28608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3ad0f920fc74e809c79af06b743b567: Bootstrap starting.
I20260924 11:39:29.236749 6992 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:39:29.237602 28608 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3ad0f920fc74e809c79af06b743b567: Neither blocks nor log segments found. Creating new log.
I20260924 11:39:29.238639 28608 log.cc:824] T 00000000000000000000000000000000 P b3ad0f920fc74e809c79af06b743b567: Log is configured to *not* fsync() on all Append() calls
I20260924 11:39:29.241993 28608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3ad0f920fc74e809c79af06b743b567: No bootstrap required, opened a new log
I20260924 11:39:29.247430 28611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:29.241147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.95:47291 (local address 127.0.0.1:39087)
0924 11:39:29.241295 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:29.241299 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:29.241770 (+ 471us) server_negotiation.cc:407] Connection header received
0924 11:39:29.241978 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:29.241986 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:29.242044 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:29.242148 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:29.242658 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:29.243497 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:29.244849 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:29.245316 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:29.245448 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:29.246635 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:29.246640 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:29.246643 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:39:29.246663 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:29.246700 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:39:29.246715 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:29.246850 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:29.247039 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 11:39:29.247091 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 11:39:29.249833 28608 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3ad0f920fc74e809c79af06b743b567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ad0f920fc74e809c79af06b743b567" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39087 } }
I20260924 11:39:29.250221 28608 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3ad0f920fc74e809c79af06b743b567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:39:29.250326 28608 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3ad0f920fc74e809c79af06b743b567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3ad0f920fc74e809c79af06b743b567, State: Initialized, Role: FOLLOWER
I20260924 11:39:29.250876 28608 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3ad0f920fc74e809c79af06b743b567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ad0f920fc74e809c79af06b743b567" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39087 } }
I20260924 11:39:29.251120 28608 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3ad0f920fc74e809c79af06b743b567 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:39:29.251263 28608 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3ad0f920fc74e809c79af06b743b567 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:39:29.251454 28608 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3ad0f920fc74e809c79af06b743b567 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:39:29.254006 28608 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3ad0f920fc74e809c79af06b743b567 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ad0f920fc74e809c79af06b743b567" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39087 } }
I20260924 11:39:29.254482 28608 leader_election.cc:302] T 00000000000000000000000000000000 P b3ad0f920fc74e809c79af06b743b567 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3ad0f920fc74e809c79af06b743b567; no voters:
I20260924 11:39:29.255021 28608 leader_election.cc:288] T 00000000000000000000000000000000 P b3ad0f920fc74e809c79af06b743b567 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:39:29.255153 28618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3ad0f920fc74e809c79af06b743b567 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:39:29.256047 28618 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3ad0f920fc74e809c79af06b743b567 [term 1 LEADER]: Becoming Leader. State: Replica: b3ad0f920fc74e809c79af06b743b567, State: Running, Role: LEADER
I20260924 11:39:29.256623 28618 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3ad0f920fc74e809c79af06b743b567 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ad0f920fc74e809c79af06b743b567" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39087 } }
I20260924 11:39:29.257236 28608 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3ad0f920fc74e809c79af06b743b567 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:39:29.259485 28620 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3ad0f920fc74e809c79af06b743b567 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3ad0f920fc74e809c79af06b743b567. Latest consensus state: current_term: 1 leader_uuid: "b3ad0f920fc74e809c79af06b743b567" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ad0f920fc74e809c79af06b743b567" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39087 } } }
I20260924 11:39:29.259843 28620 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3ad0f920fc74e809c79af06b743b567 [sys.catalog]: This master's current role is: LEADER
I20260924 11:39:29.259326 28619 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3ad0f920fc74e809c79af06b743b567 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3ad0f920fc74e809c79af06b743b567" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ad0f920fc74e809c79af06b743b567" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39087 } } }
I20260924 11:39:29.260957 28619 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3ad0f920fc74e809c79af06b743b567 [sys.catalog]: This master's current role is: LEADER
I20260924 11:39:29.262576 28626 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:39:29.267745 28626 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:39:29.276638 28626 catalog_manager.cc:1347] Generated new cluster ID: a40aa50d81464046b47e150efde29303
I20260924 11:39:29.276772 28626 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:39:29.302088 28626 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:39:29.304046 28626 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:39:29.312089 28626 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3ad0f920fc74e809c79af06b743b567: Generated new TSK 0
I20260924 11:39:29.312916 28626 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:39:29.327296 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 28541
W20260924 11:39:29.328465 28603 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.95:45395
W20260924 11:39:29.329241 6992 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 28541: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:39:29.329394 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 28541
W20260924 11:39:29.346966 28526 connection.cc:535] client connection to 127.0.0.1:39087 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39087): Connection reset by peer (error 104)
2026-09-24T11:39:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1980 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790249617057452-6992-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:39:29 dist-test-slave-cq7c krb5kdc[28639](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:39:29 dist-test-slave-cq7c krb5kdc[28639](info): set up 2 sockets
Sep 24 11:39:29 dist-test-slave-cq7c krb5kdc[28639](info): commencing operation
krb5kdc: starting...
W20260924 11:39:30.479902 6992 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.003s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Sep 24 11:39:30 dist-test-slave-cq7c krb5kdc[28639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:39:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:39:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790249617057452-6992-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790249617057452-6992-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 11:39:30.771471 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43703
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790249617057452-6992-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:39:31.116572 28652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:39:31.116936 28652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:39:31.117110 28652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:39:31.124575 28652 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:39:31.124706 28652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:39:31.124759 28652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:39:31.124831 28652 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:39:31.124882 28652 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:39:31.135337 28652 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43703
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790249617057452-6992-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:39:31.136698 28652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:39:31.138124 28652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:39:31.152273 28657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:31.152240 28658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:31.152455 28660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:39:31.154388 28652 server_base.cc:1034] running on GCE node
I20260924 11:39:31.155970 28652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:39:31.158999 28652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:39:31.160250 28652 hybrid_clock.cc:648] HybridClock initialized: now 1790249971160085 us; error 145 us; skew 500 ppm
I20260924 11:39:31.160773 28652 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:39:31 dist-test-slave-cq7c krb5kdc[28639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:39:31.165858 28652 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 11:39:31.167884 28652 webserver.cc:460] Webserver started at http://127.0.0.1:40337/ using document root <none> and password file <none>
I20260924 11:39:31.168901 28652 fs_manager.cc:359] Metadata directory not provided
I20260924 11:39:31.169065 28652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:39:31.169453 28652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:39:31.173817 28652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "be5e9a74504e4104ba6902947f485ee0"
format_stamp: "Formatted at 2026-09-24 11:39:31 on dist-test-slave-cq7c"
server_key: "9da31b6647df5fe7e65768a0227ccd1a"
server_key_iv: "660b7307ec485d4f6640478a1e783fa5"
server_key_version: "encryptionkey@0"
I20260924 11:39:31.174656 28652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "be5e9a74504e4104ba6902947f485ee0"
format_stamp: "Formatted at 2026-09-24 11:39:31 on dist-test-slave-cq7c"
server_key: "9da31b6647df5fe7e65768a0227ccd1a"
server_key_iv: "660b7307ec485d4f6640478a1e783fa5"
server_key_version: "encryptionkey@0"
I20260924 11:39:31.182994 28652 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.010s sys 0.000s
I20260924 11:39:31.187510 28667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:39:31.189698 28652 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 11:39:31.189924 28652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "be5e9a74504e4104ba6902947f485ee0"
format_stamp: "Formatted at 2026-09-24 11:39:31 on dist-test-slave-cq7c"
server_key: "9da31b6647df5fe7e65768a0227ccd1a"
server_key_iv: "660b7307ec485d4f6640478a1e783fa5"
server_key_version: "encryptionkey@0"
I20260924 11:39:31.190207 28652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:39:31.198547 28652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:39:31.204658 28652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:39:31.204998 28652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:39:31.224529 28652 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43703
I20260924 11:39:31.224663 28718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43703 every 8 connection(s)
I20260924 11:39:31.225957 28652 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:39:31.229333 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 28652
I20260924 11:39:31.229619 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:39:31.230149 28719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:39:31.230157 6992 external_mini_cluster.cc:1428] Setting key b789314c6df575cdcc7d428a0856e730
I20260924 11:39:31.239694 28722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:31.232260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.95:46035 (local address 127.0.0.1:43703)
0924 11:39:31.233094 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:31.233117 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:31.233176 (+ 59us) server_negotiation.cc:407] Connection header received
0924 11:39:31.235447 (+ 2271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:31.235477 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:31.235949 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:31.236429 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:31.237530 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:31.237540 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:31.237557 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:39:31.237576 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:31.237656 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:39:31.237691 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:31.237937 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:31.238097 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 11:39:31.238324 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":320,"threads_started":1}
I20260924 11:39:31.243856 6992 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:39:31.245755 28719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be5e9a74504e4104ba6902947f485ee0: Bootstrap starting.
I20260924 11:39:31.249454 28719 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be5e9a74504e4104ba6902947f485ee0: Neither blocks nor log segments found. Creating new log.
I20260924 11:39:31.250815 28719 log.cc:824] T 00000000000000000000000000000000 P be5e9a74504e4104ba6902947f485ee0: Log is configured to *not* fsync() on all Append() calls
I20260924 11:39:31.251674 28722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:31.248707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.95:52467 (local address 127.0.0.1:43703)
0924 11:39:31.248889 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:31.248896 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:31.249298 (+ 402us) server_negotiation.cc:407] Connection header received
0924 11:39:31.249439 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:31.249447 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:31.249518 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:31.249644 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:31.250708 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:31.250715 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:31.250719 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:39:31.250739 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:31.250796 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:39:31.250828 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:31.250975 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:31.251172 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 11:39:31.251238 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 11:39:31.254163 28719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be5e9a74504e4104ba6902947f485ee0: No bootstrap required, opened a new log
I20260924 11:39:31.262254 28719 raft_consensus.cc:348] T 00000000000000000000000000000000 P be5e9a74504e4104ba6902947f485ee0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be5e9a74504e4104ba6902947f485ee0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43703 } }
I20260924 11:39:31.262708 28719 raft_consensus.cc:374] T 00000000000000000000000000000000 P be5e9a74504e4104ba6902947f485ee0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:39:31.262825 28719 raft_consensus.cc:729] T 00000000000000000000000000000000 P be5e9a74504e4104ba6902947f485ee0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be5e9a74504e4104ba6902947f485ee0, State: Initialized, Role: FOLLOWER
I20260924 11:39:31.263614 28719 consensus_queue.cc:260] T 00000000000000000000000000000000 P be5e9a74504e4104ba6902947f485ee0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be5e9a74504e4104ba6902947f485ee0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43703 } }
I20260924 11:39:31.263934 28719 raft_consensus.cc:388] T 00000000000000000000000000000000 P be5e9a74504e4104ba6902947f485ee0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:39:31.264115 28719 raft_consensus.cc:482] T 00000000000000000000000000000000 P be5e9a74504e4104ba6902947f485ee0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:39:31.264305 28719 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be5e9a74504e4104ba6902947f485ee0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:39:31.266750 28719 raft_consensus.cc:504] T 00000000000000000000000000000000 P be5e9a74504e4104ba6902947f485ee0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be5e9a74504e4104ba6902947f485ee0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43703 } }
I20260924 11:39:31.267272 28719 leader_election.cc:302] T 00000000000000000000000000000000 P be5e9a74504e4104ba6902947f485ee0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be5e9a74504e4104ba6902947f485ee0; no voters:
I20260924 11:39:31.267951 28719 leader_election.cc:288] T 00000000000000000000000000000000 P be5e9a74504e4104ba6902947f485ee0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:39:31.268118 28729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be5e9a74504e4104ba6902947f485ee0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:39:31.269449 28729 raft_consensus.cc:686] T 00000000000000000000000000000000 P be5e9a74504e4104ba6902947f485ee0 [term 1 LEADER]: Becoming Leader. State: Replica: be5e9a74504e4104ba6902947f485ee0, State: Running, Role: LEADER
I20260924 11:39:31.270021 28729 consensus_queue.cc:237] T 00000000000000000000000000000000 P be5e9a74504e4104ba6902947f485ee0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be5e9a74504e4104ba6902947f485ee0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43703 } }
I20260924 11:39:31.270382 28719 sys_catalog.cc:564] T 00000000000000000000000000000000 P be5e9a74504e4104ba6902947f485ee0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:39:31.272775 28731 sys_catalog.cc:455] T 00000000000000000000000000000000 P be5e9a74504e4104ba6902947f485ee0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be5e9a74504e4104ba6902947f485ee0. Latest consensus state: current_term: 1 leader_uuid: "be5e9a74504e4104ba6902947f485ee0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be5e9a74504e4104ba6902947f485ee0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43703 } } }
I20260924 11:39:31.273185 28731 sys_catalog.cc:458] T 00000000000000000000000000000000 P be5e9a74504e4104ba6902947f485ee0 [sys.catalog]: This master's current role is: LEADER
I20260924 11:39:31.273748 28730 sys_catalog.cc:455] T 00000000000000000000000000000000 P be5e9a74504e4104ba6902947f485ee0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be5e9a74504e4104ba6902947f485ee0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be5e9a74504e4104ba6902947f485ee0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43703 } } }
I20260924 11:39:31.274128 28730 sys_catalog.cc:458] T 00000000000000000000000000000000 P be5e9a74504e4104ba6902947f485ee0 [sys.catalog]: This master's current role is: LEADER
I20260924 11:39:31.275401 28736 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:39:31.282426 28736 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:39:31.292019 28736 catalog_manager.cc:1347] Generated new cluster ID: 2f5acb0a3ef542598b718010f49a5f65
I20260924 11:39:31.292158 28736 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:39:31.328101 28736 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:39:31.328645 28736 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:39:31.338893 28736 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be5e9a74504e4104ba6902947f485ee0: Generated new TSK 0
I20260924 11:39:31.339836 28736 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:39:31.357833 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 28652
W20260924 11:39:31.359045 28714 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.95:46035
W20260924 11:39:31.359885 6992 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 28652: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:39:31.360077 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 28652
2026-09-24T11:39:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2031 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790249617057452-6992-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:39:31 dist-test-slave-cq7c krb5kdc[28751](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:39:31 dist-test-slave-cq7c krb5kdc[28751](info): set up 2 sockets
Sep 24 11:39:31 dist-test-slave-cq7c krb5kdc[28751](info): commencing operation
krb5kdc: starting...
W20260924 11:39:32.576514 6992 mini_kdc.cc:121] Time spent starting KDC: real 1.179s user 0.001s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Sep 24 11:39:32 dist-test-slave-cq7c krb5kdc[28751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:39:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:39:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.212.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
I20260924 11:39:32.862380 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:33967
--webserver_interface=127.6.212.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:44989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:33967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:39:33.212052 28764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:39:33.212352 28764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:39:33.212443 28764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:39:33.219904 28764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:39:33.220028 28764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:39:33.220075 28764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:39:33.220146 28764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:39:33.220191 28764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:39:33.231079 28764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:44989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:33967
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:33967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.212.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:39:33.232497 28764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:39:33.233779 28764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:39:33.246385 28769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:33.246472 28772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:33.246464 28770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:39:33.247172 28764 server_base.cc:1034] running on GCE node
I20260924 11:39:33.248097 28764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:39:33.249760 28764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:39:33.251004 28764 hybrid_clock.cc:648] HybridClock initialized: now 1790249973250955 us; error 35 us; skew 500 ppm
I20260924 11:39:33.251482 28764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:39:33 dist-test-slave-cq7c krb5kdc[28751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249973, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:39:33.254628 28764 init.cc:377] Logged in from keytab as kudu/127.6.212.62@KRBTEST.COM (short username kudu)
I20260924 11:39:33.256246 28764 webserver.cc:460] Webserver started at http://127.6.212.62:44433/ using document root <none> and password file <none>
I20260924 11:39:33.256910 28764 fs_manager.cc:359] Metadata directory not provided
I20260924 11:39:33.257009 28764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:39:33.257265 28764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:39:33.260870 28764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "6c04e2159c7d4a889986874519f4c168"
format_stamp: "Formatted at 2026-09-24 11:39:33 on dist-test-slave-cq7c"
server_key: "26a71729566135178271cb652f2a107e"
server_key_iv: "69a63a8b2d624d4ce678335f9a342bb3"
server_key_version: "encryptionkey@0"
I20260924 11:39:33.261582 28764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "6c04e2159c7d4a889986874519f4c168"
format_stamp: "Formatted at 2026-09-24 11:39:33 on dist-test-slave-cq7c"
server_key: "26a71729566135178271cb652f2a107e"
server_key_iv: "69a63a8b2d624d4ce678335f9a342bb3"
server_key_version: "encryptionkey@0"
I20260924 11:39:33.266733 28764 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 11:39:33.270329 28779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:39:33.271821 28764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 11:39:33.272013 28764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "6c04e2159c7d4a889986874519f4c168"
format_stamp: "Formatted at 2026-09-24 11:39:33 on dist-test-slave-cq7c"
server_key: "26a71729566135178271cb652f2a107e"
server_key_iv: "69a63a8b2d624d4ce678335f9a342bb3"
server_key_version: "encryptionkey@0"
I20260924 11:39:33.272241 28764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:39:33.297775 28764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:39:33.302546 28764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:39:33.302860 28764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:39:33.322091 28764 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.62:33967
I20260924 11:39:33.322114 28830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.62:33967 every 8 connection(s)
I20260924 11:39:33.323570 28764 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:39:33.327478 28831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:39:33.330636 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 28764
I20260924 11:39:33.330842 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:39:33.331293 6992 external_mini_cluster.cc:1428] Setting key 0c8d3d037c4b1f3da85be14f05003a54
I20260924 11:39:33.340354 28831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c04e2159c7d4a889986874519f4c168: Bootstrap starting.
I20260924 11:39:33.344309 28831 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c04e2159c7d4a889986874519f4c168: Neither blocks nor log segments found. Creating new log.
I20260924 11:39:33.345748 28831 log.cc:824] T 00000000000000000000000000000000 P 6c04e2159c7d4a889986874519f4c168: Log is configured to *not* fsync() on all Append() calls
Sep 24 11:39:33 dist-test-slave-cq7c krb5kdc[28751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:39:33.349263 28831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c04e2159c7d4a889986874519f4c168: No bootstrap required, opened a new log
I20260924 11:39:33.357314 28834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:33.333495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50787 (local address 127.6.212.62:33967)
0924 11:39:33.334314 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:33.334327 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:33.334373 (+ 46us) server_negotiation.cc:407] Connection header received
0924 11:39:33.336126 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:33.336150 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:33.336603 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:33.336956 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:33.338709 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:33.340181 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:33.341940 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:33.342650 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:33.342784 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:33.348250 (+ 5466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:33.348261 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:33.348283 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:39:33.348332 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:33.352292 (+ 3960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:33.353288 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:33.353296 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:33.353303 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:33.353533 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:33.354096 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:33.354102 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:33.354104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:33.354723 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:33.355155 (+ 432us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:33.355765 (+ 610us) server_negotiation.cc:299] Negotiation successful
0924 11:39:33.356146 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":399,"threads_started":1}
I20260924 11:39:33.357426 28831 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c04e2159c7d4a889986874519f4c168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c04e2159c7d4a889986874519f4c168" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 33967 } }
I20260924 11:39:33.357920 28831 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c04e2159c7d4a889986874519f4c168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:39:33.358057 28831 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c04e2159c7d4a889986874519f4c168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c04e2159c7d4a889986874519f4c168, State: Initialized, Role: FOLLOWER
I20260924 11:39:33.358698 28831 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c04e2159c7d4a889986874519f4c168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c04e2159c7d4a889986874519f4c168" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 33967 } }
I20260924 11:39:33.358994 28831 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c04e2159c7d4a889986874519f4c168 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:39:33.359165 28831 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c04e2159c7d4a889986874519f4c168 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:39:33.359416 28831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c04e2159c7d4a889986874519f4c168 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:39:33.362157 6992 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:39:33.362577 28831 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c04e2159c7d4a889986874519f4c168 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c04e2159c7d4a889986874519f4c168" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 33967 } }
I20260924 11:39:33.363119 28831 leader_election.cc:302] T 00000000000000000000000000000000 P 6c04e2159c7d4a889986874519f4c168 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c04e2159c7d4a889986874519f4c168; no voters:
I20260924 11:39:33.363790 28831 leader_election.cc:288] T 00000000000000000000000000000000 P 6c04e2159c7d4a889986874519f4c168 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:39:33.363926 28836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c04e2159c7d4a889986874519f4c168 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:39:33.365944 28831 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c04e2159c7d4a889986874519f4c168 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:39:33.372355 28836 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c04e2159c7d4a889986874519f4c168 [term 1 LEADER]: Becoming Leader. State: Replica: 6c04e2159c7d4a889986874519f4c168, State: Running, Role: LEADER
I20260924 11:39:33.373220 28836 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c04e2159c7d4a889986874519f4c168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c04e2159c7d4a889986874519f4c168" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 33967 } }
I20260924 11:39:33.376993 28838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c04e2159c7d4a889986874519f4c168 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c04e2159c7d4a889986874519f4c168. Latest consensus state: current_term: 1 leader_uuid: "6c04e2159c7d4a889986874519f4c168" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c04e2159c7d4a889986874519f4c168" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 33967 } } }
I20260924 11:39:33.377413 28838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c04e2159c7d4a889986874519f4c168 [sys.catalog]: This master's current role is: LEADER
I20260924 11:39:33.378058 28837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c04e2159c7d4a889986874519f4c168 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c04e2159c7d4a889986874519f4c168" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c04e2159c7d4a889986874519f4c168" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 33967 } } }
I20260924 11:39:33.378381 28837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c04e2159c7d4a889986874519f4c168 [sys.catalog]: This master's current role is: LEADER
I20260924 11:39:33.379997 28851 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:39:33.385979 28851 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:39:33.404256 28834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:33.375134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55457 (local address 127.6.212.62:33967)
0924 11:39:33.375435 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:33.375441 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:33.375897 (+ 456us) server_negotiation.cc:407] Connection header received
0924 11:39:33.376257 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:33.376266 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:33.376377 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:33.376474 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:33.379581 (+ 3107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:33.381213 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:33.383760 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:33.384760 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:33.384896 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:33.389327 (+ 4431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:33.389342 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:33.389351 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:39:33.389409 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:33.398193 (+ 8784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:33.399619 (+ 1426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:33.399629 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:33.399635 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:33.401701 (+ 2066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:33.402372 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:33.402382 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:33.402385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:33.402910 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:33.403206 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:33.403732 (+ 526us) server_negotiation.cc:299] Negotiation successful
0924 11:39:33.403928 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260924 11:39:33.405184 28851 catalog_manager.cc:1347] Generated new cluster ID: 0637c63fcfd149e7972f09dfbe868e5a
I20260924 11:39:33.405310 28851 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:39:33.413882 28851 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:39:33.415591 28851 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:39:33.428681 28851 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c04e2159c7d4a889986874519f4c168: Generated new TSK 0
I20260924 11:39:33.429605 28851 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:39:33.441331 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 28764
I20260924 11:39:33.442708 28826 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:39:33.536792 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 28764
W20260924 11:39:33.554906 28749 connection.cc:535] client connection to 127.6.212.62:33967 recv error: Network error: failed to read from TLS socket (remote: 127.6.212.62:33967): Connection reset by peer (error 104)
2026-09-24T11:39:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2176 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790249617057452-6992-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:39:33 dist-test-slave-cq7c krb5kdc[28864](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:39:33 dist-test-slave-cq7c krb5kdc[28864](info): set up 2 sockets
Sep 24 11:39:33 dist-test-slave-cq7c krb5kdc[28864](info): commencing operation
krb5kdc: starting...
W20260924 11:39:34.672066 6992 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Sep 24 11:39:34 dist-test-slave-cq7c krb5kdc[28864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:39:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:39:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.212.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
I20260924 11:39:34.958539 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:43787
--webserver_interface=127.6.212.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:36319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:43787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:39:35.303071 28877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:39:35.303469 28877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:39:35.303656 28877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:39:35.312103 28877 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:39:35.312278 28877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:39:35.312382 28877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:39:35.312496 28877 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:39:35.312578 28877 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:39:35.323872 28877 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:36319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:43787
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:43787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.212.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:39:35.325354 28877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:39:35.326750 28877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:39:35.339609 28882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:35.339586 28883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:35.339639 28885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:39:35.340637 28877 server_base.cc:1034] running on GCE node
I20260924 11:39:35.341434 28877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:39:35.342861 28877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:39:35.344120 28877 hybrid_clock.cc:648] HybridClock initialized: now 1790249975344056 us; error 62 us; skew 500 ppm
I20260924 11:39:35.344589 28877 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:39:35 dist-test-slave-cq7c krb5kdc[28864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249975, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:39:35.348141 28877 init.cc:377] Logged in from keytab as kudu/127.6.212.62@KRBTEST.COM (short username kudu)
I20260924 11:39:35.349867 28877 webserver.cc:460] Webserver started at http://127.6.212.62:44369/ using document root <none> and password file <none>
I20260924 11:39:35.350939 28877 fs_manager.cc:359] Metadata directory not provided
I20260924 11:39:35.351109 28877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:39:35.351533 28877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:39:35.354808 28877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "62875959e15c4eed9a7c9596cb72056c"
format_stamp: "Formatted at 2026-09-24 11:39:35 on dist-test-slave-cq7c"
server_key: "480ff59cd7eebadb7e09323b93e25adf"
server_key_iv: "e69a30acced62ef2ac762d10d67ea89a"
server_key_version: "encryptionkey@0"
I20260924 11:39:35.355718 28877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "62875959e15c4eed9a7c9596cb72056c"
format_stamp: "Formatted at 2026-09-24 11:39:35 on dist-test-slave-cq7c"
server_key: "480ff59cd7eebadb7e09323b93e25adf"
server_key_iv: "e69a30acced62ef2ac762d10d67ea89a"
server_key_version: "encryptionkey@0"
I20260924 11:39:35.361348 28877 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 11:39:35.365157 28892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:39:35.366652 28877 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 11:39:35.366801 28877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "62875959e15c4eed9a7c9596cb72056c"
format_stamp: "Formatted at 2026-09-24 11:39:35 on dist-test-slave-cq7c"
server_key: "480ff59cd7eebadb7e09323b93e25adf"
server_key_iv: "e69a30acced62ef2ac762d10d67ea89a"
server_key_version: "encryptionkey@0"
I20260924 11:39:35.366992 28877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:39:35.390566 28877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:39:35.395331 28877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:39:35.395704 28877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:39:35.416643 28877 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.62:43787
I20260924 11:39:35.416738 28943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.62:43787 every 8 connection(s)
I20260924 11:39:35.418123 28877 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:39:35.422034 28944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:39:35.426631 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 28877
I20260924 11:39:35.426801 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:39:35.427240 6992 external_mini_cluster.cc:1428] Setting key 6225dfb6fdc490f154231811b9c870f5
I20260924 11:39:35.433298 28944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62875959e15c4eed9a7c9596cb72056c: Bootstrap starting.
I20260924 11:39:35.437091 28944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62875959e15c4eed9a7c9596cb72056c: Neither blocks nor log segments found. Creating new log.
I20260924 11:39:35.438446 28944 log.cc:824] T 00000000000000000000000000000000 P 62875959e15c4eed9a7c9596cb72056c: Log is configured to *not* fsync() on all Append() calls
I20260924 11:39:35.441861 28944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62875959e15c4eed9a7c9596cb72056c: No bootstrap required, opened a new log
I20260924 11:39:35.442112 28947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:35.429276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47237 (local address 127.6.212.62:43787)
0924 11:39:35.430142 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:35.430155 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:35.430196 (+ 41us) server_negotiation.cc:407] Connection header received
0924 11:39:35.432004 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:35.432029 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:35.432435 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:35.432773 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:35.433451 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:35.434835 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:35.436752 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:35.437616 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:35.437753 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:35.439413 (+ 1660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:35.439425 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:35.439447 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:39:35.440256 (+ 809us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:35.440330 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:39:35.440356 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:35.440491 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:35.440856 (+ 365us) server_negotiation.cc:299] Negotiation successful
0924 11:39:35.441063 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":432,"threads_started":1}
I20260924 11:39:35.445952 6992 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:39:35.450207 28944 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62875959e15c4eed9a7c9596cb72056c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62875959e15c4eed9a7c9596cb72056c" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43787 } }
I20260924 11:39:35.450719 28944 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62875959e15c4eed9a7c9596cb72056c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:39:35.450847 28944 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62875959e15c4eed9a7c9596cb72056c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62875959e15c4eed9a7c9596cb72056c, State: Initialized, Role: FOLLOWER
I20260924 11:39:35.451574 28944 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62875959e15c4eed9a7c9596cb72056c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62875959e15c4eed9a7c9596cb72056c" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43787 } }
I20260924 11:39:35.451833 28944 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62875959e15c4eed9a7c9596cb72056c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:39:35.452008 28944 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62875959e15c4eed9a7c9596cb72056c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:39:35.452198 28944 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62875959e15c4eed9a7c9596cb72056c [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:39:35.454952 28944 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62875959e15c4eed9a7c9596cb72056c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62875959e15c4eed9a7c9596cb72056c" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43787 } }
I20260924 11:39:35.455677 28944 leader_election.cc:302] T 00000000000000000000000000000000 P 62875959e15c4eed9a7c9596cb72056c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62875959e15c4eed9a7c9596cb72056c; no voters:
I20260924 11:39:35.456497 28944 leader_election.cc:288] T 00000000000000000000000000000000 P 62875959e15c4eed9a7c9596cb72056c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:39:35.456622 28954 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62875959e15c4eed9a7c9596cb72056c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:39:35.457036 28947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:35.450172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39913 (local address 127.6.212.62:43787)
0924 11:39:35.450393 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:35.450402 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:35.450904 (+ 502us) server_negotiation.cc:407] Connection header received
0924 11:39:35.451092 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:35.451096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:35.451153 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:35.451264 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:35.451779 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:35.452700 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:35.454015 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:35.454494 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:35.454623 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:35.456070 (+ 1447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:35.456074 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:35.456077 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:39:35.456096 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:35.456131 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:39:35.456145 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:35.456276 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:35.456482 (+ 206us) server_negotiation.cc:299] Negotiation successful
0924 11:39:35.456619 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 11:39:35.459537 28944 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62875959e15c4eed9a7c9596cb72056c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:39:35.461097 28954 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62875959e15c4eed9a7c9596cb72056c [term 1 LEADER]: Becoming Leader. State: Replica: 62875959e15c4eed9a7c9596cb72056c, State: Running, Role: LEADER
I20260924 11:39:35.461961 28954 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62875959e15c4eed9a7c9596cb72056c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62875959e15c4eed9a7c9596cb72056c" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43787 } }
I20260924 11:39:35.465478 28956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62875959e15c4eed9a7c9596cb72056c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62875959e15c4eed9a7c9596cb72056c. Latest consensus state: current_term: 1 leader_uuid: "62875959e15c4eed9a7c9596cb72056c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62875959e15c4eed9a7c9596cb72056c" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43787 } } }
I20260924 11:39:35.465870 28956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62875959e15c4eed9a7c9596cb72056c [sys.catalog]: This master's current role is: LEADER
I20260924 11:39:35.465600 28955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62875959e15c4eed9a7c9596cb72056c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62875959e15c4eed9a7c9596cb72056c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62875959e15c4eed9a7c9596cb72056c" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43787 } } }
I20260924 11:39:35.466053 28955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62875959e15c4eed9a7c9596cb72056c [sys.catalog]: This master's current role is: LEADER
I20260924 11:39:35.467803 28964 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:39:35.472260 28964 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:39:35.481403 28964 catalog_manager.cc:1347] Generated new cluster ID: 7dc3d2e3a42545ac8e0b707377352555
I20260924 11:39:35.481534 28964 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:39:35.500671 28964 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:39:35.502647 28964 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:39:35.513731 28964 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62875959e15c4eed9a7c9596cb72056c: Generated new TSK 0
I20260924 11:39:35.514806 28964 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:39:35.523594 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 28877
W20260924 11:39:35.524673 28939 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47237
W20260924 11:39:35.525409 6992 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 28877: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:39:35.525548 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 28877
W20260924 11:39:35.543682 28862 connection.cc:535] client connection to 127.6.212.62:43787 recv error: Network error: failed to read from TLS socket (remote: 127.6.212.62:43787): Connection reset by peer (error 104)
2026-09-24T11:39:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1988 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790249617057452-6992-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:39:35 dist-test-slave-cq7c krb5kdc[28975](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:39:35 dist-test-slave-cq7c krb5kdc[28975](info): set up 2 sockets
Sep 24 11:39:35 dist-test-slave-cq7c krb5kdc[28975](info): commencing operation
krb5kdc: starting...
W20260924 11:39:36.635987 6992 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Sep 24 11:39:36 dist-test-slave-cq7c krb5kdc[28975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:39:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:39:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.212.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
I20260924 11:39:36.914889 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:36681
--webserver_interface=127.6.212.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:33327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:36681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:39:37.260139 28988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:39:37.260473 28988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:39:37.260650 28988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:39:37.268061 28988 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:39:37.268222 28988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:39:37.268344 28988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:39:37.268455 28988 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:39:37.268568 28988 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:39:37.278504 28988 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:33327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:36681
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:36681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.212.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:39:37.279888 28988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:39:37.281255 28988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:39:37.294200 28994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:37.294304 28996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:37.294744 28993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:39:37.295092 28988 server_base.cc:1034] running on GCE node
I20260924 11:39:37.296047 28988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:39:37.297672 28988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:39:37.298981 28988 hybrid_clock.cc:648] HybridClock initialized: now 1790249977298911 us; error 52 us; skew 500 ppm
I20260924 11:39:37.299539 28988 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:39:37 dist-test-slave-cq7c krb5kdc[28975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249977, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:39:37.306836 28988 init.cc:377] Logged in from keytab as kudu/127.6.212.62@KRBTEST.COM (short username kudu)
I20260924 11:39:37.308597 28988 webserver.cc:460] Webserver started at http://127.6.212.62:38331/ using document root <none> and password file <none>
I20260924 11:39:37.309345 28988 fs_manager.cc:359] Metadata directory not provided
I20260924 11:39:37.309521 28988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:39:37.309849 28988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:39:37.313011 28988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "c7b9aef4989a4ad695efeebfbccab5aa"
format_stamp: "Formatted at 2026-09-24 11:39:37 on dist-test-slave-cq7c"
server_key: "b39c6c47913a98bfe87e9c926f2b422e"
server_key_iv: "c8ac92c0c9d69e61cae37249c4f3d506"
server_key_version: "encryptionkey@0"
I20260924 11:39:37.313803 28988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "c7b9aef4989a4ad695efeebfbccab5aa"
format_stamp: "Formatted at 2026-09-24 11:39:37 on dist-test-slave-cq7c"
server_key: "b39c6c47913a98bfe87e9c926f2b422e"
server_key_iv: "c8ac92c0c9d69e61cae37249c4f3d506"
server_key_version: "encryptionkey@0"
I20260924 11:39:37.319463 28988 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 11:39:37.323139 29003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:39:37.324647 28988 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 11:39:37.324837 28988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "c7b9aef4989a4ad695efeebfbccab5aa"
format_stamp: "Formatted at 2026-09-24 11:39:37 on dist-test-slave-cq7c"
server_key: "b39c6c47913a98bfe87e9c926f2b422e"
server_key_iv: "c8ac92c0c9d69e61cae37249c4f3d506"
server_key_version: "encryptionkey@0"
I20260924 11:39:37.325029 28988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:39:37.330915 28988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:39:37.335279 28988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:39:37.335623 28988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:39:37.354172 28988 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.62:36681
I20260924 11:39:37.354214 29054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.62:36681 every 8 connection(s)
I20260924 11:39:37.355634 28988 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:39:37.359628 29055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:39:37.361935 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 28988
I20260924 11:39:37.362111 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:39:37.362681 6992 external_mini_cluster.cc:1428] Setting key 99b6466dbb10b295c254b6b845016804
I20260924 11:39:37.370391 29058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:37.364481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57297 (local address 127.6.212.62:36681)
0924 11:39:37.365311 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:37.365325 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:37.365365 (+ 40us) server_negotiation.cc:407] Connection header received
0924 11:39:37.367040 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:37.367061 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:37.367478 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:37.367797 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:37.368829 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:37.368835 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:37.368851 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:39:37.368865 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:37.368927 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:39:37.368948 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:37.369122 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:37.369226 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 11:39:37.369428 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":385,"threads_started":1}
I20260924 11:39:37.371968 29055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7b9aef4989a4ad695efeebfbccab5aa: Bootstrap starting.
I20260924 11:39:37.373917 6992 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:39:37.375422 29055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7b9aef4989a4ad695efeebfbccab5aa: Neither blocks nor log segments found. Creating new log.
I20260924 11:39:37.376698 29055 log.cc:824] T 00000000000000000000000000000000 P c7b9aef4989a4ad695efeebfbccab5aa: Log is configured to *not* fsync() on all Append() calls
I20260924 11:39:37.379933 29055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7b9aef4989a4ad695efeebfbccab5aa: No bootstrap required, opened a new log
I20260924 11:39:37.382079 29058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:37.379230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45121 (local address 127.6.212.62:36681)
0924 11:39:37.379446 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:37.379453 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:37.379983 (+ 530us) server_negotiation.cc:407] Connection header received
0924 11:39:37.380212 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:37.380217 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:37.380280 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:37.380393 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:37.381391 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:37.381398 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:37.381402 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:39:37.381418 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:37.381461 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:39:37.381481 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:37.381606 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:37.381680 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 11:39:37.381740 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 11:39:37.388214 29055 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7b9aef4989a4ad695efeebfbccab5aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7b9aef4989a4ad695efeebfbccab5aa" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36681 } }
I20260924 11:39:37.388557 29055 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7b9aef4989a4ad695efeebfbccab5aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:39:37.388630 29055 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7b9aef4989a4ad695efeebfbccab5aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7b9aef4989a4ad695efeebfbccab5aa, State: Initialized, Role: FOLLOWER
I20260924 11:39:37.389115 29055 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7b9aef4989a4ad695efeebfbccab5aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7b9aef4989a4ad695efeebfbccab5aa" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36681 } }
I20260924 11:39:37.389389 29055 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7b9aef4989a4ad695efeebfbccab5aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:39:37.389566 29055 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7b9aef4989a4ad695efeebfbccab5aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:39:37.389780 29055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7b9aef4989a4ad695efeebfbccab5aa [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:39:37.392009 29055 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7b9aef4989a4ad695efeebfbccab5aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7b9aef4989a4ad695efeebfbccab5aa" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36681 } }
I20260924 11:39:37.392565 29055 leader_election.cc:302] T 00000000000000000000000000000000 P c7b9aef4989a4ad695efeebfbccab5aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7b9aef4989a4ad695efeebfbccab5aa; no voters:
I20260924 11:39:37.393222 29055 leader_election.cc:288] T 00000000000000000000000000000000 P c7b9aef4989a4ad695efeebfbccab5aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:39:37.393354 29065 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7b9aef4989a4ad695efeebfbccab5aa [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:39:37.394040 29065 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7b9aef4989a4ad695efeebfbccab5aa [term 1 LEADER]: Becoming Leader. State: Replica: c7b9aef4989a4ad695efeebfbccab5aa, State: Running, Role: LEADER
I20260924 11:39:37.394598 29065 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7b9aef4989a4ad695efeebfbccab5aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7b9aef4989a4ad695efeebfbccab5aa" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36681 } }
I20260924 11:39:37.395327 29055 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7b9aef4989a4ad695efeebfbccab5aa [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:39:37.397665 29067 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7b9aef4989a4ad695efeebfbccab5aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7b9aef4989a4ad695efeebfbccab5aa. Latest consensus state: current_term: 1 leader_uuid: "c7b9aef4989a4ad695efeebfbccab5aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7b9aef4989a4ad695efeebfbccab5aa" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36681 } } }
I20260924 11:39:37.397760 29066 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7b9aef4989a4ad695efeebfbccab5aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7b9aef4989a4ad695efeebfbccab5aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7b9aef4989a4ad695efeebfbccab5aa" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36681 } } }
I20260924 11:39:37.398214 29067 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7b9aef4989a4ad695efeebfbccab5aa [sys.catalog]: This master's current role is: LEADER
I20260924 11:39:37.398234 29066 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7b9aef4989a4ad695efeebfbccab5aa [sys.catalog]: This master's current role is: LEADER
I20260924 11:39:37.400019 29074 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:39:37.404584 29074 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:39:37.412539 29074 catalog_manager.cc:1347] Generated new cluster ID: 20df36bc2d624110886c4138926638f8
I20260924 11:39:37.412686 29074 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:39:37.448063 29074 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:39:37.448520 29074 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:39:37.465335 29074 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7b9aef4989a4ad695efeebfbccab5aa: Generated new TSK 0
I20260924 11:39:37.466408 29074 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:39:37.477751 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 28988
W20260924 11:39:37.478883 29050 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57297
W20260924 11:39:37.479666 6992 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 28988: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:39:37.479846 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 28988
2026-09-24T11:39:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1951 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790249617057452-6992-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:39:37 dist-test-slave-cq7c krb5kdc[29086](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:39:37 dist-test-slave-cq7c krb5kdc[29086](info): set up 2 sockets
Sep 24 11:39:37 dist-test-slave-cq7c krb5kdc[29086](info): commencing operation
krb5kdc: starting...
W20260924 11:39:38.599864 6992 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.006s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Sep 24 11:39:38 dist-test-slave-cq7c krb5kdc[29086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:39:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:39:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.212.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
I20260924 11:39:38.875555 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:43705
--webserver_interface=127.6.212.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:35961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:43705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:39:39.213883 29099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:39:39.214208 29099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:39:39.214308 29099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:39:39.221534 29099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:39:39.221647 29099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:39:39.221693 29099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:39:39.221755 29099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:39:39.221797 29099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:39:39.231299 29099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:35961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:43705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:43705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.212.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:39:39.232561 29099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:39:39.233819 29099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:39:39.245473 29104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:39.245466 29105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:39.245368 29107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:39:39.246243 29099 server_base.cc:1034] running on GCE node
I20260924 11:39:39.247000 29099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:39:39.248422 29099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:39:39.249657 29099 hybrid_clock.cc:648] HybridClock initialized: now 1790249979249594 us; error 55 us; skew 500 ppm
I20260924 11:39:39.250054 29099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:39:39 dist-test-slave-cq7c krb5kdc[29086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249979, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:39:39.253281 29099 init.cc:377] Logged in from keytab as kudu/127.6.212.62@KRBTEST.COM (short username kudu)
I20260924 11:39:39.254742 29099 webserver.cc:460] Webserver started at http://127.6.212.62:36413/ using document root <none> and password file <none>
I20260924 11:39:39.255493 29099 fs_manager.cc:359] Metadata directory not provided
I20260924 11:39:39.255636 29099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:39:39.255964 29099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:39:39.258807 29099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "c67e31227f014635b496e7f8b92840cc"
format_stamp: "Formatted at 2026-09-24 11:39:39 on dist-test-slave-cq7c"
server_key: "ed1cd6792777181638d7f6fb8b11b2f3"
server_key_iv: "00c73b4b56ba0deab4604cde64856d7a"
server_key_version: "encryptionkey@0"
I20260924 11:39:39.259645 29099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "c67e31227f014635b496e7f8b92840cc"
format_stamp: "Formatted at 2026-09-24 11:39:39 on dist-test-slave-cq7c"
server_key: "ed1cd6792777181638d7f6fb8b11b2f3"
server_key_iv: "00c73b4b56ba0deab4604cde64856d7a"
server_key_version: "encryptionkey@0"
I20260924 11:39:39.264617 29099 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 11:39:39.267894 29114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:39:39.269222 29099 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:39:39.269425 29099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "c67e31227f014635b496e7f8b92840cc"
format_stamp: "Formatted at 2026-09-24 11:39:39 on dist-test-slave-cq7c"
server_key: "ed1cd6792777181638d7f6fb8b11b2f3"
server_key_iv: "00c73b4b56ba0deab4604cde64856d7a"
server_key_version: "encryptionkey@0"
I20260924 11:39:39.269654 29099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:39:39.296950 29099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:39:39.301651 29099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:39:39.301956 29099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:39:39.321225 29099 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.62:43705
I20260924 11:39:39.321275 29165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.62:43705 every 8 connection(s)
I20260924 11:39:39.322676 29099 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:39:39.323227 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 29099
I20260924 11:39:39.323446 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:39:39.324101 6992 external_mini_cluster.cc:1428] Setting key c736fc530d5d323c12fddcd1a13b98d9
I20260924 11:39:39.327090 29166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 11:39:39 dist-test-slave-cq7c krb5kdc[29086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:39:39.340806 29166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c67e31227f014635b496e7f8b92840cc: Bootstrap starting.
I20260924 11:39:39.344141 29166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c67e31227f014635b496e7f8b92840cc: Neither blocks nor log segments found. Creating new log.
I20260924 11:39:39.345191 29166 log.cc:824] T 00000000000000000000000000000000 P c67e31227f014635b496e7f8b92840cc: Log is configured to *not* fsync() on all Append() calls
I20260924 11:39:39.348255 29166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c67e31227f014635b496e7f8b92840cc: No bootstrap required, opened a new log
I20260924 11:39:39.349014 29169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:39.326099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53369 (local address 127.6.212.62:43705)
0924 11:39:39.326940 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:39.326953 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:39.326997 (+ 44us) server_negotiation.cc:407] Connection header received
0924 11:39:39.328825 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:39.328848 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:39.329257 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:39.329627 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:39.331453 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:39.333006 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:39.334848 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:39.335748 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:39.335894 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:39.341051 (+ 5157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:39.341062 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:39.341079 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:39:39.341118 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:39.344625 (+ 3507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:39.345375 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:39.345388 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:39.345396 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:39.345567 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:39.346003 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:39.346009 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:39.346011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:39.346591 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:39.346915 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:39.347436 (+ 521us) server_negotiation.cc:299] Negotiation successful
0924 11:39:39.347888 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":389,"threads_started":1}
I20260924 11:39:39.353147 6992 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:39:39.356884 29166 raft_consensus.cc:348] T 00000000000000000000000000000000 P c67e31227f014635b496e7f8b92840cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67e31227f014635b496e7f8b92840cc" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43705 } }
I20260924 11:39:39.357312 29166 raft_consensus.cc:374] T 00000000000000000000000000000000 P c67e31227f014635b496e7f8b92840cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:39:39.357470 29166 raft_consensus.cc:729] T 00000000000000000000000000000000 P c67e31227f014635b496e7f8b92840cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c67e31227f014635b496e7f8b92840cc, State: Initialized, Role: FOLLOWER
I20260924 11:39:39.358242 29166 consensus_queue.cc:260] T 00000000000000000000000000000000 P c67e31227f014635b496e7f8b92840cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67e31227f014635b496e7f8b92840cc" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43705 } }
I20260924 11:39:39.358487 29166 raft_consensus.cc:388] T 00000000000000000000000000000000 P c67e31227f014635b496e7f8b92840cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:39:39.358654 29166 raft_consensus.cc:482] T 00000000000000000000000000000000 P c67e31227f014635b496e7f8b92840cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:39:39.358843 29166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c67e31227f014635b496e7f8b92840cc [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:39:39.361218 29166 raft_consensus.cc:504] T 00000000000000000000000000000000 P c67e31227f014635b496e7f8b92840cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67e31227f014635b496e7f8b92840cc" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43705 } }
I20260924 11:39:39.361701 29166 leader_election.cc:302] T 00000000000000000000000000000000 P c67e31227f014635b496e7f8b92840cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c67e31227f014635b496e7f8b92840cc; no voters:
I20260924 11:39:39.362378 29166 leader_election.cc:288] T 00000000000000000000000000000000 P c67e31227f014635b496e7f8b92840cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:39:39.362478 29176 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c67e31227f014635b496e7f8b92840cc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:39:39.363673 29176 raft_consensus.cc:686] T 00000000000000000000000000000000 P c67e31227f014635b496e7f8b92840cc [term 1 LEADER]: Becoming Leader. State: Replica: c67e31227f014635b496e7f8b92840cc, State: Running, Role: LEADER
I20260924 11:39:39.364274 29176 consensus_queue.cc:237] T 00000000000000000000000000000000 P c67e31227f014635b496e7f8b92840cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67e31227f014635b496e7f8b92840cc" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43705 } }
I20260924 11:39:39.364835 29166 sys_catalog.cc:564] T 00000000000000000000000000000000 P c67e31227f014635b496e7f8b92840cc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:39:39.367213 29177 sys_catalog.cc:455] T 00000000000000000000000000000000 P c67e31227f014635b496e7f8b92840cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c67e31227f014635b496e7f8b92840cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67e31227f014635b496e7f8b92840cc" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43705 } } }
I20260924 11:39:39.367722 29177 sys_catalog.cc:458] T 00000000000000000000000000000000 P c67e31227f014635b496e7f8b92840cc [sys.catalog]: This master's current role is: LEADER
I20260924 11:39:39.369335 29178 sys_catalog.cc:455] T 00000000000000000000000000000000 P c67e31227f014635b496e7f8b92840cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader c67e31227f014635b496e7f8b92840cc. Latest consensus state: current_term: 1 leader_uuid: "c67e31227f014635b496e7f8b92840cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67e31227f014635b496e7f8b92840cc" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43705 } } }
I20260924 11:39:39.369688 29178 sys_catalog.cc:458] T 00000000000000000000000000000000 P c67e31227f014635b496e7f8b92840cc [sys.catalog]: This master's current role is: LEADER
I20260924 11:39:39.371436 29181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:39:39.376214 29169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:39.357625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33725 (local address 127.6.212.62:43705)
0924 11:39:39.357806 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:39.357815 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:39.358276 (+ 461us) server_negotiation.cc:407] Connection header received
0924 11:39:39.358518 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:39.358523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:39.358585 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:39.358709 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:39.360399 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:39.361224 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:39.362731 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:39.363554 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:39.363715 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:39.367022 (+ 3307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:39.367028 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:39.367036 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:39:39.367083 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:39.371681 (+ 4598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:39.372712 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:39.372719 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:39.372726 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:39.372955 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:39.373518 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:39.373526 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:39.373531 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:39.374387 (+ 856us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:39.374719 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:39.375556 (+ 837us) server_negotiation.cc:299] Negotiation successful
0924 11:39:39.375798 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 11:39:39.377940 29181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:39:39.387348 29181 catalog_manager.cc:1347] Generated new cluster ID: 25279e46bfc14088bab699d62ab82432
I20260924 11:39:39.387552 29181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:39:39.402712 29181 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:39:39.404486 29181 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:39:39.412571 29181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c67e31227f014635b496e7f8b92840cc: Generated new TSK 0
I20260924 11:39:39.413654 29181 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:39:39.423895 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 29099
I20260924 11:39:39.425335 29161 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:39:39.515767 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 29099
2026-09-24T11:39:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2040 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790249617057452-6992-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:39:39 dist-test-slave-cq7c krb5kdc[29198](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:39:39 dist-test-slave-cq7c krb5kdc[29198](info): set up 2 sockets
Sep 24 11:39:39 dist-test-slave-cq7c krb5kdc[29198](info): commencing operation
krb5kdc: starting...
W20260924 11:39:40.631918 6992 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.008s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Sep 24 11:39:40 dist-test-slave-cq7c krb5kdc[29198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:39:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:39:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.212.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
I20260924 11:39:40.909006 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:35517
--webserver_interface=127.6.212.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:43973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:35517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:39:41.273947 29211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:39:41.274273 29211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:39:41.274453 29211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:39:41.282824 29211 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:39:41.282964 29211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:39:41.283087 29211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:39:41.283221 29211 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:39:41.283311 29211 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:39:41.294836 29211 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:43973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:35517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:35517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.212.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:39:41.296357 29211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:39:41.298367 29211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:39:41.311568 29219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:41.311658 29216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:41.311682 29217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:39:41.312021 29211 server_base.cc:1034] running on GCE node
I20260924 11:39:41.313182 29211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:39:41.314697 29211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:39:41.315943 29211 hybrid_clock.cc:648] HybridClock initialized: now 1790249981315873 us; error 59 us; skew 500 ppm
I20260924 11:39:41.316325 29211 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:39:41 dist-test-slave-cq7c krb5kdc[29198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249981, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:39:41.319758 29211 init.cc:377] Logged in from keytab as kudu/127.6.212.62@KRBTEST.COM (short username kudu)
I20260924 11:39:41.321278 29211 webserver.cc:460] Webserver started at http://127.6.212.62:39923/ using document root <none> and password file <none>
I20260924 11:39:41.321960 29211 fs_manager.cc:359] Metadata directory not provided
I20260924 11:39:41.322070 29211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:39:41.322324 29211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:39:41.325768 29211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "09110b77b0ca492fae89f66788cc8fa0"
format_stamp: "Formatted at 2026-09-24 11:39:41 on dist-test-slave-cq7c"
server_key: "cb3962280feae7b652e64a8cf8020a8d"
server_key_iv: "dd2b7f2eb012a26a36e2831a7df912b1"
server_key_version: "encryptionkey@0"
I20260924 11:39:41.326527 29211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "09110b77b0ca492fae89f66788cc8fa0"
format_stamp: "Formatted at 2026-09-24 11:39:41 on dist-test-slave-cq7c"
server_key: "cb3962280feae7b652e64a8cf8020a8d"
server_key_iv: "dd2b7f2eb012a26a36e2831a7df912b1"
server_key_version: "encryptionkey@0"
I20260924 11:39:41.332191 29211 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 11:39:41.335955 29226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:39:41.337409 29211 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260924 11:39:41.337605 29211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "09110b77b0ca492fae89f66788cc8fa0"
format_stamp: "Formatted at 2026-09-24 11:39:41 on dist-test-slave-cq7c"
server_key: "cb3962280feae7b652e64a8cf8020a8d"
server_key_iv: "dd2b7f2eb012a26a36e2831a7df912b1"
server_key_version: "encryptionkey@0"
I20260924 11:39:41.337787 29211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:39:41.365583 29211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:39:41.370167 29211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:39:41.370424 29211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:39:41.389766 29211 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.62:35517
I20260924 11:39:41.389816 29277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.62:35517 every 8 connection(s)
I20260924 11:39:41.391225 29211 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:39:41.395153 29278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:39:41.396904 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 29211
I20260924 11:39:41.397099 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:39:41.397701 6992 external_mini_cluster.cc:1428] Setting key e113480225c0cd9c78cc60a6d22820a7
I20260924 11:39:41.410331 29278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09110b77b0ca492fae89f66788cc8fa0: Bootstrap starting.
I20260924 11:39:41.412168 29281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:41.399780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60357 (local address 127.6.212.62:35517)
0924 11:39:41.400690 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:41.400705 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:41.400759 (+ 54us) server_negotiation.cc:407] Connection header received
0924 11:39:41.402462 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:41.402485 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:41.402956 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:41.403310 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:41.404021 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:41.405346 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:41.407468 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:41.408202 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:41.408332 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:41.409738 (+ 1406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:41.409745 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:41.409759 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:39:41.410296 (+ 537us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:41.410378 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:39:41.410402 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:41.410542 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:41.410791 (+ 249us) server_negotiation.cc:299] Negotiation successful
0924 11:39:41.411111 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":425,"threads_started":1}
I20260924 11:39:41.413961 29278 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09110b77b0ca492fae89f66788cc8fa0: Neither blocks nor log segments found. Creating new log.
I20260924 11:39:41.415215 29278 log.cc:824] T 00000000000000000000000000000000 P 09110b77b0ca492fae89f66788cc8fa0: Log is configured to *not* fsync() on all Append() calls
I20260924 11:39:41.415951 6992 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:39:41.418264 29278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09110b77b0ca492fae89f66788cc8fa0: No bootstrap required, opened a new log
I20260924 11:39:41.426451 29281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:41.420349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46719 (local address 127.6.212.62:35517)
0924 11:39:41.420522 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:41.420529 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:41.420951 (+ 422us) server_negotiation.cc:407] Connection header received
0924 11:39:41.421162 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:41.421166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:41.421231 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:41.421351 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:41.421856 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:41.422584 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:41.423942 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:41.424415 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:41.424563 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:41.425748 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:41.425752 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:41.425755 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:39:41.425771 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:41.425808 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:39:41.425826 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:41.425933 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:41.426064 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 11:39:41.426154 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 11:39:41.427476 29278 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09110b77b0ca492fae89f66788cc8fa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09110b77b0ca492fae89f66788cc8fa0" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 35517 } }
I20260924 11:39:41.427891 29278 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09110b77b0ca492fae89f66788cc8fa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:39:41.428036 29278 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09110b77b0ca492fae89f66788cc8fa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09110b77b0ca492fae89f66788cc8fa0, State: Initialized, Role: FOLLOWER
I20260924 11:39:41.428630 29278 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09110b77b0ca492fae89f66788cc8fa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09110b77b0ca492fae89f66788cc8fa0" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 35517 } }
I20260924 11:39:41.428907 29278 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09110b77b0ca492fae89f66788cc8fa0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:39:41.429072 29278 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09110b77b0ca492fae89f66788cc8fa0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:39:41.429261 29278 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09110b77b0ca492fae89f66788cc8fa0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:39:41.431676 29278 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09110b77b0ca492fae89f66788cc8fa0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09110b77b0ca492fae89f66788cc8fa0" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 35517 } }
I20260924 11:39:41.432364 29278 leader_election.cc:302] T 00000000000000000000000000000000 P 09110b77b0ca492fae89f66788cc8fa0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09110b77b0ca492fae89f66788cc8fa0; no voters:
I20260924 11:39:41.433007 29278 leader_election.cc:288] T 00000000000000000000000000000000 P 09110b77b0ca492fae89f66788cc8fa0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:39:41.433104 29288 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09110b77b0ca492fae89f66788cc8fa0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:39:41.433930 29288 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09110b77b0ca492fae89f66788cc8fa0 [term 1 LEADER]: Becoming Leader. State: Replica: 09110b77b0ca492fae89f66788cc8fa0, State: Running, Role: LEADER
I20260924 11:39:41.434511 29288 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09110b77b0ca492fae89f66788cc8fa0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09110b77b0ca492fae89f66788cc8fa0" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 35517 } }
I20260924 11:39:41.435477 29278 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09110b77b0ca492fae89f66788cc8fa0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:39:41.437314 29290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09110b77b0ca492fae89f66788cc8fa0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09110b77b0ca492fae89f66788cc8fa0. Latest consensus state: current_term: 1 leader_uuid: "09110b77b0ca492fae89f66788cc8fa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09110b77b0ca492fae89f66788cc8fa0" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 35517 } } }
I20260924 11:39:41.437638 29289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09110b77b0ca492fae89f66788cc8fa0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09110b77b0ca492fae89f66788cc8fa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09110b77b0ca492fae89f66788cc8fa0" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 35517 } } }
I20260924 11:39:41.437878 29290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09110b77b0ca492fae89f66788cc8fa0 [sys.catalog]: This master's current role is: LEADER
I20260924 11:39:41.437878 29289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09110b77b0ca492fae89f66788cc8fa0 [sys.catalog]: This master's current role is: LEADER
I20260924 11:39:41.439631 29294 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:39:41.446195 29294 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:39:41.455333 29294 catalog_manager.cc:1347] Generated new cluster ID: 1e362138a8df4db0a287ac14c34140b5
I20260924 11:39:41.455623 29294 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:39:41.467399 29294 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:39:41.469318 29294 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:39:41.481875 29294 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09110b77b0ca492fae89f66788cc8fa0: Generated new TSK 0
I20260924 11:39:41.483022 29294 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:39:41.495630 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 29211
W20260924 11:39:41.496824 29273 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60357
W20260924 11:39:41.497641 6992 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 29211: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:39:41.497797 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 29211
2026-09-24T11:39:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1979 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790249617057452-6992-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:39:41 dist-test-slave-cq7c krb5kdc[29309](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:39:41 dist-test-slave-cq7c krb5kdc[29309](info): set up 2 sockets
Sep 24 11:39:41 dist-test-slave-cq7c krb5kdc[29309](info): commencing operation
krb5kdc: starting...
W20260924 11:39:42.655927 6992 mini_kdc.cc:121] Time spent starting KDC: real 1.121s user 0.002s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Sep 24 11:39:42 dist-test-slave-cq7c krb5kdc[29309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:39:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:39:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.212.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
I20260924 11:39:42.955731 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:36953
--webserver_interface=127.6.212.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:35251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:36953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:39:43.293174 29322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:39:43.293463 29322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:39:43.293624 29322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:39:43.300755 29322 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:39:43.300863 29322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:39:43.301004 29322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:39:43.301091 29322 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:39:43.301190 29322 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:39:43.310967 29322 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:35251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:36953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:36953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.212.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:39:43.312264 29322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:39:43.313609 29322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:39:43.325610 29327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:43.325610 29328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:43.325670 29330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:39:43.326421 29322 server_base.cc:1034] running on GCE node
I20260924 11:39:43.327189 29322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:39:43.328644 29322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:39:43.329842 29322 hybrid_clock.cc:648] HybridClock initialized: now 1790249983329805 us; error 32 us; skew 500 ppm
I20260924 11:39:43.330214 29322 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:39:43 dist-test-slave-cq7c krb5kdc[29309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249983, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:39:43.333519 29322 init.cc:377] Logged in from keytab as kudu/127.6.212.62@KRBTEST.COM (short username kudu)
I20260924 11:39:43.335060 29322 webserver.cc:460] Webserver started at http://127.6.212.62:34949/ using document root <none> and password file <none>
I20260924 11:39:43.335811 29322 fs_manager.cc:359] Metadata directory not provided
I20260924 11:39:43.335956 29322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:39:43.336273 29322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:39:43.339243 29322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "f44ac1bf4d8c45cc8fb160024ee5128d"
format_stamp: "Formatted at 2026-09-24 11:39:43 on dist-test-slave-cq7c"
server_key: "a2765948ab6402aa09121f03c7d0504d"
server_key_iv: "d0ba068745e77760d6099794db5d3cd7"
server_key_version: "encryptionkey@0"
I20260924 11:39:43.340061 29322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "f44ac1bf4d8c45cc8fb160024ee5128d"
format_stamp: "Formatted at 2026-09-24 11:39:43 on dist-test-slave-cq7c"
server_key: "a2765948ab6402aa09121f03c7d0504d"
server_key_iv: "d0ba068745e77760d6099794db5d3cd7"
server_key_version: "encryptionkey@0"
I20260924 11:39:43.345289 29322 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 11:39:43.348840 29337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:39:43.350116 29322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 11:39:43.350296 29322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "f44ac1bf4d8c45cc8fb160024ee5128d"
format_stamp: "Formatted at 2026-09-24 11:39:43 on dist-test-slave-cq7c"
server_key: "a2765948ab6402aa09121f03c7d0504d"
server_key_iv: "d0ba068745e77760d6099794db5d3cd7"
server_key_version: "encryptionkey@0"
I20260924 11:39:43.350497 29322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:39:43.356144 29322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:39:43.360365 29322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:39:43.360654 29322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:39:43.380263 29322 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.62:36953
I20260924 11:39:43.380293 29388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.62:36953 every 8 connection(s)
I20260924 11:39:43.381780 29322 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:39:43.382203 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 29322
I20260924 11:39:43.382352 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:39:43.382843 6992 external_mini_cluster.cc:1428] Setting key 885c7362814e288023383529edfa7a67
I20260924 11:39:43.386570 29389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:39:43.390581 29392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:43.384818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34457 (local address 127.6.212.62:36953)
0924 11:39:43.385620 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:43.385631 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:43.385677 (+ 46us) server_negotiation.cc:407] Connection header received
0924 11:39:43.387293 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:43.387315 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:43.387688 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:43.388049 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:43.389049 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:43.389055 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:43.389067 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:39:43.389090 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:43.389159 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:39:43.389181 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:43.389406 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:43.389555 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 11:39:43.389718 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":352,"threads_started":1}
I20260924 11:39:43.394400 6992 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:39:43.398939 29389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f44ac1bf4d8c45cc8fb160024ee5128d: Bootstrap starting.
I20260924 11:39:43.401252 29392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:43.398584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43827 (local address 127.6.212.62:36953)
0924 11:39:43.398725 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:43.398729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:43.399294 (+ 565us) server_negotiation.cc:407] Connection header received
0924 11:39:43.399564 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:43.399569 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:43.399620 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:43.399715 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:43.400628 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:43.400634 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:43.400638 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 11:39:43.400664 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:43.400702 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 11:39:43.400716 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:43.400830 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:43.400921 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 11:39:43.400977 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 11:39:43.402572 29389 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f44ac1bf4d8c45cc8fb160024ee5128d: Neither blocks nor log segments found. Creating new log.
I20260924 11:39:43.403633 29389 log.cc:824] T 00000000000000000000000000000000 P f44ac1bf4d8c45cc8fb160024ee5128d: Log is configured to *not* fsync() on all Append() calls
I20260924 11:39:43.406320 29389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f44ac1bf4d8c45cc8fb160024ee5128d: No bootstrap required, opened a new log
I20260924 11:39:43.414222 29389 raft_consensus.cc:348] T 00000000000000000000000000000000 P f44ac1bf4d8c45cc8fb160024ee5128d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f44ac1bf4d8c45cc8fb160024ee5128d" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36953 } }
I20260924 11:39:43.414723 29389 raft_consensus.cc:374] T 00000000000000000000000000000000 P f44ac1bf4d8c45cc8fb160024ee5128d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 11:39:43.414857 29389 raft_consensus.cc:729] T 00000000000000000000000000000000 P f44ac1bf4d8c45cc8fb160024ee5128d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f44ac1bf4d8c45cc8fb160024ee5128d, State: Initialized, Role: FOLLOWER
I20260924 11:39:43.415613 29389 consensus_queue.cc:260] T 00000000000000000000000000000000 P f44ac1bf4d8c45cc8fb160024ee5128d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f44ac1bf4d8c45cc8fb160024ee5128d" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36953 } }
I20260924 11:39:43.415863 29389 raft_consensus.cc:388] T 00000000000000000000000000000000 P f44ac1bf4d8c45cc8fb160024ee5128d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 11:39:43.416021 29389 raft_consensus.cc:482] T 00000000000000000000000000000000 P f44ac1bf4d8c45cc8fb160024ee5128d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 11:39:43.416190 29389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f44ac1bf4d8c45cc8fb160024ee5128d [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:39:43.418848 29389 raft_consensus.cc:504] T 00000000000000000000000000000000 P f44ac1bf4d8c45cc8fb160024ee5128d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f44ac1bf4d8c45cc8fb160024ee5128d" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36953 } }
I20260924 11:39:43.419420 29389 leader_election.cc:302] T 00000000000000000000000000000000 P f44ac1bf4d8c45cc8fb160024ee5128d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f44ac1bf4d8c45cc8fb160024ee5128d; no voters:
I20260924 11:39:43.420145 29389 leader_election.cc:288] T 00000000000000000000000000000000 P f44ac1bf4d8c45cc8fb160024ee5128d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 11:39:43.420253 29399 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f44ac1bf4d8c45cc8fb160024ee5128d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:39:43.421084 29399 raft_consensus.cc:686] T 00000000000000000000000000000000 P f44ac1bf4d8c45cc8fb160024ee5128d [term 1 LEADER]: Becoming Leader. State: Replica: f44ac1bf4d8c45cc8fb160024ee5128d, State: Running, Role: LEADER
I20260924 11:39:43.421619 29399 consensus_queue.cc:237] T 00000000000000000000000000000000 P f44ac1bf4d8c45cc8fb160024ee5128d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f44ac1bf4d8c45cc8fb160024ee5128d" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36953 } }
I20260924 11:39:43.422250 29389 sys_catalog.cc:564] T 00000000000000000000000000000000 P f44ac1bf4d8c45cc8fb160024ee5128d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:39:43.424588 29400 sys_catalog.cc:455] T 00000000000000000000000000000000 P f44ac1bf4d8c45cc8fb160024ee5128d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f44ac1bf4d8c45cc8fb160024ee5128d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f44ac1bf4d8c45cc8fb160024ee5128d" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36953 } } }
I20260924 11:39:43.424598 29401 sys_catalog.cc:455] T 00000000000000000000000000000000 P f44ac1bf4d8c45cc8fb160024ee5128d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f44ac1bf4d8c45cc8fb160024ee5128d. Latest consensus state: current_term: 1 leader_uuid: "f44ac1bf4d8c45cc8fb160024ee5128d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f44ac1bf4d8c45cc8fb160024ee5128d" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36953 } } }
I20260924 11:39:43.425130 29401 sys_catalog.cc:458] T 00000000000000000000000000000000 P f44ac1bf4d8c45cc8fb160024ee5128d [sys.catalog]: This master's current role is: LEADER
I20260924 11:39:43.425132 29400 sys_catalog.cc:458] T 00000000000000000000000000000000 P f44ac1bf4d8c45cc8fb160024ee5128d [sys.catalog]: This master's current role is: LEADER
I20260924 11:39:43.426834 29405 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:39:43.432893 29405 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:39:43.441938 29405 catalog_manager.cc:1347] Generated new cluster ID: f62d649703eb4b7eb0dd24272ea98e76
I20260924 11:39:43.442073 29405 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:39:43.465546 29405 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:39:43.466076 29405 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:39:43.473316 29405 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f44ac1bf4d8c45cc8fb160024ee5128d: Generated new TSK 0
I20260924 11:39:43.474310 29405 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:39:43.487666 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 29322
W20260924 11:39:43.489252 29384 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34457
W20260924 11:39:43.489940 6992 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 29322: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 11:39:43.490104 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 29322
2026-09-24T11:39:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1991 ms)
[----------] 18 tests from AuthTokenIssuingTest (36421 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:39:43 dist-test-slave-cq7c krb5kdc[29420](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:39:43 dist-test-slave-cq7c krb5kdc[29420](info): set up 2 sockets
Sep 24 11:39:43 dist-test-slave-cq7c krb5kdc[29420](info): commencing operation
krb5kdc: starting...
W20260924 11:39:44.616110 6992 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.004s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Sep 24 11:39:44 dist-test-slave-cq7c krb5kdc[29420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:39:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:39:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.212.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
I20260924 11:39:44.911500 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:43257
--webserver_interface=127.6.212.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:45021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:43257,127.6.212.61:45173,127.6.212.60:33835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:39:45.251788 29433 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 11:39:45.252156 29433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:39:45.252283 29433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:39:45.252415 29433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:39:45.260433 29433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:39:45.260528 29433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:39:45.260622 29433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:39:45.260694 29433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:39:45.260779 29433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:39:45.270638 29433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:45021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:43257,127.6.212.61:45173,127.6.212.60:33835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:43257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.212.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:39:45.272025 29433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:39:45.273309 29433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:39:45.285107 29441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:45.285207 29438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:45.285176 29439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:39:45.285943 29433 server_base.cc:1034] running on GCE node
I20260924 11:39:45.286695 29433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:39:45.288173 29433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:39:45.289388 29433 hybrid_clock.cc:648] HybridClock initialized: now 1790249985289333 us; error 36 us; skew 500 ppm
I20260924 11:39:45.289690 29433 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:39:45 dist-test-slave-cq7c krb5kdc[29420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249985, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:39:45.292963 29433 init.cc:377] Logged in from keytab as kudu/127.6.212.62@KRBTEST.COM (short username kudu)
I20260924 11:39:45.294468 29433 webserver.cc:460] Webserver started at http://127.6.212.62:40449/ using document root <none> and password file <none>
I20260924 11:39:45.295212 29433 fs_manager.cc:359] Metadata directory not provided
I20260924 11:39:45.295339 29433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:39:45.295750 29433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:39:45.299072 29433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "653450b0060b4696be946b14f122d9dd"
format_stamp: "Formatted at 2026-09-24 11:39:45 on dist-test-slave-cq7c"
server_key: "96eb6065e9ee1b6c5c970d9bfc828f8e"
server_key_iv: "bf41d0ee22b74d107ac2e647568f674b"
server_key_version: "encryptionkey@0"
I20260924 11:39:45.300068 29433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "653450b0060b4696be946b14f122d9dd"
format_stamp: "Formatted at 2026-09-24 11:39:45 on dist-test-slave-cq7c"
server_key: "96eb6065e9ee1b6c5c970d9bfc828f8e"
server_key_iv: "bf41d0ee22b74d107ac2e647568f674b"
server_key_version: "encryptionkey@0"
I20260924 11:39:45.305860 29433 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 11:39:45.309617 29448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:39:45.311070 29433 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 11:39:45.311286 29433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "653450b0060b4696be946b14f122d9dd"
format_stamp: "Formatted at 2026-09-24 11:39:45 on dist-test-slave-cq7c"
server_key: "96eb6065e9ee1b6c5c970d9bfc828f8e"
server_key_iv: "bf41d0ee22b74d107ac2e647568f674b"
server_key_version: "encryptionkey@0"
I20260924 11:39:45.311662 29433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:39:45.329161 29433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:39:45.333539 29433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:39:45.333828 29433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:39:45.354370 29433 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.62:43257
I20260924 11:39:45.354390 29499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.62:43257 every 8 connection(s)
I20260924 11:39:45.355870 29433 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:39:45.359640 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 29433
I20260924 11:39:45.359859 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:39:45.360049 29500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:39:45.360714 6992 external_mini_cluster.cc:1428] Setting key bcc14a4fc3c4314676bd27b1d6a8a5a4
I20260924 11:39:45.370260 29500 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43257 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:39:45 dist-test-slave-cq7c krb5kdc[29420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249985, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.212.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.61@KRBTEST.COM" created.
I20260924 11:39:45.396827 29503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:45.374371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41302 (local address 127.6.212.62:43257)
0924 11:39:45.375282 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:45.375300 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:45.375908 (+ 608us) server_negotiation.cc:407] Connection header received
0924 11:39:45.377914 (+ 2006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:45.377925 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:45.378273 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:45.378502 (+ 229us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:45.379975 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:45.381957 (+ 1982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:45.383526 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:45.384494 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:45.384736 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:45.388724 (+ 3988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:45.388735 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:45.388753 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:39:45.388801 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:45.392280 (+ 3479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:45.392943 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:45.392949 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:45.392952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:45.393101 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:45.393463 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:45.393466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:45.393468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:45.394157 (+ 689us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:45.394318 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:45.395035 (+ 717us) server_negotiation.cc:299] Negotiation successful
0924 11:39:45.395316 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":644,"thread_start_us":398,"threads_started":1}
I20260924 11:39:45.396899 29504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:45.374371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:43257 (local address 127.0.0.1:41302)
0924 11:39:45.375717 (+ 1346us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:45.375766 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 11:39:45.377534 (+ 1768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:39:45.378693 (+ 1159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:39:45.378699 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:39:45.379324 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:39:45.379751 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:45.379769 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:45.382146 (+ 2377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:45.382150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:45.383302 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:45.383314 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:45.384762 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:45.384769 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:45.384934 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:45.385722 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:39:45.385754 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:39:45.388455 (+ 2701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:39:45.392404 (+ 3949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:39:45.392413 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:39:45.392422 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:39:45.392816 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:39:45.393204 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:39:45.393207 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:39:45.393209 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:39:45.393364 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:39:45.394308 (+ 944us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:39:45.394318 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:39:45.394579 (+ 261us) client_negotiation.cc:769] Sending connection context
0924 11:39:45.394853 (+ 274us) client_negotiation.cc:240] Negotiation successful
0924 11:39:45.395510 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1075,"mutex_wait_us":73,"thread_start_us":869,"threads_started":1}
I20260924 11:39:45.400153 29500 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.212.61" port: 45173 } has no permanent_uuid. Determining permanent_uuid...
I20260924 11:39:45.401400 29504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:45.400900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:45173 (local address 127.0.0.1:44542)
0924 11:39:45.401106 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:45.401240 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.61:45173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260924 11:39:45.401703 29449 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.212.61:45173: connect: Connection refused (error 111)
I20260924 11:39:45.403414 29504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:45.402962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:45173 (local address 127.0.0.1:44558)
0924 11:39:45.403134 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:45.403226 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.61:45173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 11:39:45.403650 29500 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.212.61:45173: Network error: Client connection negotiation failed: client connection to 127.6.212.61:45173: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.61.keytab.
Entry for principal kudu/127.6.212.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.61.keytab.
I20260924 11:39:45.430841 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.61:45173
--webserver_interface=127.6.212.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.61.keytab
--principal=kudu/127.6.212.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:45021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:43257,127.6.212.61:45173,127.6.212.60:33835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:39:45.453025 29500 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.212.61" port: 45173 } attempt: 1
I20260924 11:39:45.454289 29504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:45.453839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:45173 (local address 127.0.0.1:44574)
0924 11:39:45.454055 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:45.454144 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.61:45173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260924 11:39:45.455263 29504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:45.454918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:45173 (local address 127.0.0.1:44586)
0924 11:39:45.455081 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:45.455144 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.61:45173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 11:39:45.455462 29500 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.212.61:45173: Network error: Client connection negotiation failed: client connection to 127.6.212.61:45173: connect: Connection refused (error 111)
I20260924 11:39:45.523730 29500 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.212.61" port: 45173 } attempt: 2
I20260924 11:39:45.525071 29504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:45.524541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:45173 (local address 127.0.0.1:44592)
0924 11:39:45.524820 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:45.524921 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.61:45173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260924 11:39:45.526330 29504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:45.525770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:45173 (local address 127.0.0.1:44606)
0924 11:39:45.526007 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:45.526133 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.61:45173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260924 11:39:45.526611 29500 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.212.61:45173: Network error: Client connection negotiation failed: client connection to 127.6.212.61:45173: connect: Connection refused (error 111)
I20260924 11:39:45.617870 29500 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.212.61" port: 45173 } attempt: 3
I20260924 11:39:45.619050 29504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:45.618627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:45173 (local address 127.0.0.1:44614)
0924 11:39:45.618818 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:45.618905 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.61:45173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 11:39:45.620154 29504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:45.619805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:45173 (local address 127.0.0.1:44630)
0924 11:39:45.619942 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:45.620019 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.61:45173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 11:39:45.620416 29500 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.212.61:45173: Network error: Client connection negotiation failed: client connection to 127.6.212.61:45173: connect: Connection refused (error 111)
I20260924 11:39:45.763751 29500 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.212.61" port: 45173 } attempt: 4
I20260924 11:39:45.765407 29504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:45.764717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:45173 (local address 127.0.0.1:44638)
0924 11:39:45.765164 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:45.765259 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.61:45173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298}
I20260924 11:39:45.766590 29504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:45.766199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:45173 (local address 127.0.0.1:44646)
0924 11:39:45.766360 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:45.766461 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.61:45173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 11:39:45.766790 29500 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.212.61:45173: Network error: Client connection negotiation failed: client connection to 127.6.212.61:45173: connect: Connection refused (error 111)
W20260924 11:39:45.772482 29507 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 11:39:45.772863 29507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:39:45.772987 29507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:39:45.773119 29507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:39:45.780704 29507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:39:45.780836 29507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:39:45.780965 29507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:39:45.781051 29507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:39:45.781159 29507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:39:45.791819 29507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:45021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:43257,127.6.212.61:45173,127.6.212.60:33835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.61.keytab
--principal=kudu/127.6.212.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.61:45173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.212.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:39:45.793239 29507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:39:45.794607 29507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:39:45.807039 29515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:45.807076 29513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:45.807126 29512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:39:45.808722 29507 server_base.cc:1034] running on GCE node
I20260924 11:39:45.809720 29507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:39:45.811406 29507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:39:45.812700 29507 hybrid_clock.cc:648] HybridClock initialized: now 1790249985812632 us; error 44 us; skew 500 ppm
I20260924 11:39:45.813083 29507 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:39:45 dist-test-slave-cq7c krb5kdc[29420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249985, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:39:45.816879 29507 init.cc:377] Logged in from keytab as kudu/127.6.212.61@KRBTEST.COM (short username kudu)
I20260924 11:39:45.818446 29507 webserver.cc:460] Webserver started at http://127.6.212.61:39187/ using document root <none> and password file <none>
I20260924 11:39:45.819154 29507 fs_manager.cc:359] Metadata directory not provided
I20260924 11:39:45.819242 29507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:39:45.819589 29507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:39:45.822647 29507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-1/data/instance:
uuid: "d4b83c0ee5094dfe941227c6cedaf071"
format_stamp: "Formatted at 2026-09-24 11:39:45 on dist-test-slave-cq7c"
server_key: "f88dffc0a9684542e10bc60544c0b7a1"
server_key_iv: "5b1652ac234b500257f563778c53062b"
server_key_version: "encryptionkey@0"
I20260924 11:39:45.823479 29507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-1/wal/instance:
uuid: "d4b83c0ee5094dfe941227c6cedaf071"
format_stamp: "Formatted at 2026-09-24 11:39:45 on dist-test-slave-cq7c"
server_key: "f88dffc0a9684542e10bc60544c0b7a1"
server_key_iv: "5b1652ac234b500257f563778c53062b"
server_key_version: "encryptionkey@0"
I20260924 11:39:45.828861 29507 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 11:39:45.832633 29522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:39:45.834158 29507 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:39:45.834304 29507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-1/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-1/wal
uuid: "d4b83c0ee5094dfe941227c6cedaf071"
format_stamp: "Formatted at 2026-09-24 11:39:45 on dist-test-slave-cq7c"
server_key: "f88dffc0a9684542e10bc60544c0b7a1"
server_key_iv: "5b1652ac234b500257f563778c53062b"
server_key_version: "encryptionkey@0"
I20260924 11:39:45.834496 29507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:39:45.849185 29507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:39:45.854388 29507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:39:45.854624 29507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:39:45.875262 29507 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.61:45173
I20260924 11:39:45.875295 29573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.61:45173 every 8 connection(s)
I20260924 11:39:45.877552 29507 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-1/data/info.pb
I20260924 11:39:45.878309 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 29507
I20260924 11:39:45.878523 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-1/wal/instance
I20260924 11:39:45.879133 6992 external_mini_cluster.cc:1428] Setting key d2a7d5ea83426f68cb21ec2f6eea9d8b
I20260924 11:39:45.883944 29574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:39:45.891888 29574 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43257 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:39:45 dist-test-slave-cq7c krb5kdc[29420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249985, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.61@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.212.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.60@KRBTEST.COM" created.
I20260924 11:39:45.920043 29503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:45.895551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41318 (local address 127.6.212.62:43257)
0924 11:39:45.895807 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:45.895816 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:45.897202 (+ 1386us) server_negotiation.cc:407] Connection header received
0924 11:39:45.900263 (+ 3061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:45.900268 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:45.900358 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:45.900473 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:45.902795 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:45.903671 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:45.906354 (+ 2683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:45.906947 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:45.907061 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:45.912019 (+ 4958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:45.912025 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:45.912028 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:39:45.912067 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:45.916024 (+ 3957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:45.916996 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:45.917006 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:45.917014 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:45.917194 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:45.917681 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:45.917689 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:45.917696 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:45.918165 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:45.918335 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:45.919238 (+ 903us) server_negotiation.cc:299] Negotiation successful
0924 11:39:45.919573 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 11:39:45.920789 29577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:45.896127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:43257 (local address 127.0.0.1:41318)
0924 11:39:45.896997 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:45.897062 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 11:39:45.899916 (+ 2854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:39:45.900772 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:39:45.900786 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:39:45.901650 (+ 864us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:39:45.902583 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:45.902611 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:45.903839 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:45.903847 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:45.906145 (+ 2298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:45.906165 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:45.907200 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:45.907217 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:45.907684 (+ 467us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:45.908488 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:39:45.908527 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:39:45.911751 (+ 3224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:39:45.916187 (+ 4436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:39:45.916195 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:39:45.916210 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:39:45.916833 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:39:45.917351 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:39:45.917355 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:39:45.917357 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:39:45.917555 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:39:45.918299 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:39:45.918305 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:39:45.918739 (+ 434us) client_negotiation.cc:769] Sending connection context
0924 11:39:45.919010 (+ 271us) client_negotiation.cc:240] Negotiation successful
0924 11:39:45.919558 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":335,"threads_started":1}
I20260924 11:39:45.922564 29574 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.212.61" port: 45173 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:39:45 dist-test-slave-cq7c krb5kdc[29420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249985, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.61@KRBTEST.COM for kudu/127.6.212.61@KRBTEST.COM
I20260924 11:39:45.944298 29577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:45.923514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:45173 (local address 127.0.0.1:44656)
0924 11:39:45.923762 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:45.923786 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 11:39:45.924001 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:39:45.927924 (+ 3923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:39:45.927932 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:39:45.928474 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:39:45.928632 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:45.928641 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:45.931047 (+ 2406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:45.931051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:45.932026 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:45.932037 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:45.933397 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:45.933402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:45.933550 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:45.934215 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:39:45.934232 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:39:45.936320 (+ 2088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:39:45.941499 (+ 5179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:39:45.941503 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:39:45.941506 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:39:45.941886 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:39:45.942489 (+ 603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:39:45.942493 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:39:45.942496 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:39:45.942624 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:39:45.943549 (+ 925us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:39:45.943553 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:39:45.943696 (+ 143us) client_negotiation.cc:769] Sending connection context
0924 11:39:45.943808 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 11:39:45.944027 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 11:39:45.944965 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:45.923694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44656 (local address 127.6.212.61:45173)
0924 11:39:45.927305 (+ 3611us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:45.927314 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:45.927351 (+ 37us) server_negotiation.cc:407] Connection header received
0924 11:39:45.927459 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:45.927465 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:45.927720 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:45.927870 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:45.928847 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:45.930887 (+ 2040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:45.932174 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:45.933247 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:45.933402 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:45.936567 (+ 3165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:45.936584 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:45.936605 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:39:45.936651 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:45.941298 (+ 4647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:45.942051 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:45.942072 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:45.942079 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:45.942258 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:45.942777 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:45.942783 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:45.942790 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:45.943396 (+ 606us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:45.943664 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:45.944054 (+ 390us) server_negotiation.cc:299] Negotiation successful
0924 11:39:45.944398 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3381,"thread_start_us":357,"threads_started":1}
I20260924 11:39:45.947579 29574 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.212.60" port: 33835 } has no permanent_uuid. Determining permanent_uuid...
I20260924 11:39:45.949007 29577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:45.948446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:33835 (local address 127.0.0.1:55564)
0924 11:39:45.948704 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:45.948833 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:33835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
W20260924 11:39:45.949363 29525 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.212.60:33835: connect: Connection refused (error 111)
I20260924 11:39:45.951083 29577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:45.950714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:33835 (local address 127.0.0.1:55568)
0924 11:39:45.950888 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:45.950971 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:33835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260924 11:39:45.951534 29574 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.212.60:33835: Network error: Client connection negotiation failed: client connection to 127.6.212.60:33835: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.60.keytab.
Entry for principal kudu/127.6.212.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.60.keytab.
I20260924 11:39:45.958796 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.60:33835
--webserver_interface=127.6.212.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.60.keytab
--principal=kudu/127.6.212.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:45021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:43257,127.6.212.61:45173,127.6.212.60:33835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:39:46.000897 29574 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.212.60" port: 33835 } attempt: 1
I20260924 11:39:46.002012 29577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:46.001560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:33835 (local address 127.0.0.1:55576)
0924 11:39:46.001752 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:46.001858 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:33835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260924 11:39:46.003055 29577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:46.002693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:33835 (local address 127.0.0.1:55584)
0924 11:39:46.002841 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:46.002928 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:33835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 11:39:46.003253 29574 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.212.60:33835: Network error: Client connection negotiation failed: client connection to 127.6.212.60:33835: connect: Connection refused (error 111)
I20260924 11:39:46.066107 29500 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.212.61" port: 45173 } attempt: 5
I20260924 11:39:46.071496 29574 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.212.60" port: 33835 } attempt: 2
I20260924 11:39:46.072680 29577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:46.072127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:33835 (local address 127.0.0.1:55590)
0924 11:39:46.072415 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:46.072506 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:33835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
I20260924 11:39:46.074039 29577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:46.073657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:33835 (local address 127.0.0.1:55602)
0924 11:39:46.073829 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:46.073913 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:33835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260924 11:39:46.074361 29574 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.212.60:33835: Network error: Client connection negotiation failed: client connection to 127.6.212.60:33835: connect: Connection refused (error 111)
Sep 24 11:39:46 dist-test-slave-cq7c krb5kdc[29420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249985, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for kudu/127.6.212.61@KRBTEST.COM
I20260924 11:39:46.082716 29504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:46.066884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:45173 (local address 127.0.0.1:44662)
0924 11:39:46.067128 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:46.067151 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 11:39:46.067285 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:39:46.067742 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:39:46.067746 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:39:46.068275 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:39:46.068513 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:46.068525 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:46.069665 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:46.069671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:46.071056 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:46.071071 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:46.072231 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:46.072240 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:46.072524 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:46.073219 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:39:46.073244 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:39:46.075792 (+ 2548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:39:46.079284 (+ 3492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:39:46.079290 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:39:46.079294 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:39:46.079910 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:39:46.080971 (+ 1061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:39:46.080976 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:39:46.080980 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:39:46.081139 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:39:46.081704 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:39:46.081709 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:39:46.081879 (+ 170us) client_negotiation.cc:769] Sending connection context
0924 11:39:46.082050 (+ 171us) client_negotiation.cc:240] Negotiation successful
0924 11:39:46.082365 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260924 11:39:46.082715 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:46.067162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44662 (local address 127.6.212.61:45173)
0924 11:39:46.067352 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:46.067404 (+ 52us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:46.067429 (+ 25us) server_negotiation.cc:407] Connection header received
0924 11:39:46.067492 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:46.067496 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:46.067575 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:46.067667 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:46.068683 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:46.069517 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:46.071230 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:46.072052 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:46.072192 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:46.075988 (+ 3796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:46.075994 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:46.075997 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:39:46.076030 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:46.079122 (+ 3092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:46.080681 (+ 1559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:46.080686 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:46.080688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:46.080845 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:46.081266 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:46.081272 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:46.081275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:46.081584 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:46.081736 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:46.082149 (+ 413us) server_negotiation.cc:299] Negotiation successful
0924 11:39:46.082378 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 11:39:46.083945 29500 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.212.60" port: 33835 } has no permanent_uuid. Determining permanent_uuid...
I20260924 11:39:46.084959 29504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:46.084592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:33835 (local address 127.0.0.1:55610)
0924 11:39:46.084762 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:46.084832 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:33835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 11:39:46.085955 29504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:46.085629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:33835 (local address 127.0.0.1:55616)
0924 11:39:46.085784 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:46.085848 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:33835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260924 11:39:46.086236 29500 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.212.60:33835: Network error: Client connection negotiation failed: client connection to 127.6.212.60:33835: connect: Connection refused (error 111)
I20260924 11:39:46.137480 29500 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.212.60" port: 33835 } attempt: 1
I20260924 11:39:46.138752 29504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:46.138221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:33835 (local address 127.0.0.1:55626)
0924 11:39:46.138410 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:46.138583 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:33835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 11:39:46.139863 29504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:46.139493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:33835 (local address 127.0.0.1:55638)
0924 11:39:46.139645 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:46.139730 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:33835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 11:39:46.140123 29500 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.212.60:33835: Network error: Client connection negotiation failed: client connection to 127.6.212.60:33835: connect: Connection refused (error 111)
I20260924 11:39:46.165618 29574 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.212.60" port: 33835 } attempt: 3
I20260924 11:39:46.166801 29577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:46.166281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:33835 (local address 127.0.0.1:55652)
0924 11:39:46.166553 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:46.166647 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:33835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 11:39:46.167850 29577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:46.167462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:33835 (local address 127.0.0.1:55660)
0924 11:39:46.167681 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:46.167743 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:33835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
W20260924 11:39:46.168053 29574 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.212.60:33835: Network error: Client connection negotiation failed: client connection to 127.6.212.60:33835: connect: Connection refused (error 111)
I20260924 11:39:46.208375 29500 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.212.60" port: 33835 } attempt: 2
I20260924 11:39:46.209648 29504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:46.209023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:33835 (local address 127.0.0.1:55662)
0924 11:39:46.209277 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:46.209420 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:33835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260924 11:39:46.210758 29504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:46.210330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:33835 (local address 127.0.0.1:55678)
0924 11:39:46.210506 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:46.210607 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:33835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 11:39:46.211015 29500 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.212.60:33835: Network error: Client connection negotiation failed: client connection to 127.6.212.60:33835: connect: Connection refused (error 111)
W20260924 11:39:46.308961 29581 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 11:39:46.309324 29581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:39:46.309458 29581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:39:46.309552 29581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 11:39:46.311283 29574 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.212.60" port: 33835 } attempt: 4
I20260924 11:39:46.312765 29577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:46.312126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:33835 (local address 127.0.0.1:33542)
0924 11:39:46.312387 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:46.312585 (+ 198us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:33835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260924 11:39:46.313930 29577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:46.313545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:33835 (local address 127.0.0.1:33554)
0924 11:39:46.313736 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:46.313817 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:33835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 11:39:46.314199 29574 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.212.60:33835: Network error: Client connection negotiation failed: client connection to 127.6.212.60:33835: connect: Connection refused (error 111)
I20260924 11:39:46.317232 29500 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.212.60" port: 33835 } attempt: 3
W20260924 11:39:46.317755 29581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:39:46.317862 29581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:39:46.317960 29581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:39:46.318044 29581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:39:46.318137 29581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:39:46.318286 29504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:46.317759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:33835 (local address 127.0.0.1:33558)
0924 11:39:46.318002 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:46.318108 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:33835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 11:39:46.319518 29504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:46.319052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:33835 (local address 127.0.0.1:33566)
0924 11:39:46.319265 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:46.319355 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:33835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260924 11:39:46.319779 29500 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.212.60:33835: Network error: Client connection negotiation failed: client connection to 127.6.212.60:33835: connect: Connection refused (error 111)
I20260924 11:39:46.328799 29581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:45021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:43257,127.6.212.61:45173,127.6.212.60:33835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.60.keytab
--principal=kudu/127.6.212.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.60:33835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.212.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:39:46.330114 29581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:39:46.331487 29581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:39:46.343699 29587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:46.343760 29589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:46.343737 29586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:39:46.344592 29581 server_base.cc:1034] running on GCE node
I20260924 11:39:46.345327 29581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:39:46.346791 29581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:39:46.348090 29581 hybrid_clock.cc:648] HybridClock initialized: now 1790249986348006 us; error 57 us; skew 500 ppm
I20260924 11:39:46.348453 29581 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:39:46 dist-test-slave-cq7c krb5kdc[29420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249986, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:39:46.352203 29581 init.cc:377] Logged in from keytab as kudu/127.6.212.60@KRBTEST.COM (short username kudu)
I20260924 11:39:46.353875 29581 webserver.cc:460] Webserver started at http://127.6.212.60:46763/ using document root <none> and password file <none>
I20260924 11:39:46.354818 29581 fs_manager.cc:359] Metadata directory not provided
I20260924 11:39:46.354974 29581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:39:46.355404 29581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:39:46.359715 29581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-2/data/instance:
uuid: "d275d51b647043f687b57837ff3220df"
format_stamp: "Formatted at 2026-09-24 11:39:46 on dist-test-slave-cq7c"
server_key: "715b5195212c247baf3596f2ad9dc0f4"
server_key_iv: "e3259256e58379a8e828ee6902aa9549"
server_key_version: "encryptionkey@0"
I20260924 11:39:46.360805 29581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-2/wal/instance:
uuid: "d275d51b647043f687b57837ff3220df"
format_stamp: "Formatted at 2026-09-24 11:39:46 on dist-test-slave-cq7c"
server_key: "715b5195212c247baf3596f2ad9dc0f4"
server_key_iv: "e3259256e58379a8e828ee6902aa9549"
server_key_version: "encryptionkey@0"
I20260924 11:39:46.367722 29581 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260924 11:39:46.371312 29596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:39:46.372749 29581 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 11:39:46.372895 29581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-2/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-2/wal
uuid: "d275d51b647043f687b57837ff3220df"
format_stamp: "Formatted at 2026-09-24 11:39:46 on dist-test-slave-cq7c"
server_key: "715b5195212c247baf3596f2ad9dc0f4"
server_key_iv: "e3259256e58379a8e828ee6902aa9549"
server_key_version: "encryptionkey@0"
I20260924 11:39:46.373093 29581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:39:46.404728 29581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:39:46.409116 29581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:39:46.409371 29581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:39:46.430778 29581 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.60:33835
I20260924 11:39:46.430814 29647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.60:33835 every 8 connection(s)
I20260924 11:39:46.432471 29581 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-2/data/info.pb
I20260924 11:39:46.436457 29648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:39:46.437393 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 29581
I20260924 11:39:46.437592 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790249617057452-6992-0/minicluster-data/master-2/wal/instance
I20260924 11:39:46.438041 6992 external_mini_cluster.cc:1428] Setting key 5b717bbf0b060e51851fbcd887b7eade
I20260924 11:39:46.446084 29648 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43257 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:39:46 dist-test-slave-cq7c krb5kdc[29420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.60@KRBTEST.COM
Sep 24 11:39:46 dist-test-slave-cq7c krb5kdc[29420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249986, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.60@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:39:46.463208 29651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:46.439980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33568 (local address 127.6.212.60:33835)
0924 11:39:46.440738 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:46.440750 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:46.440789 (+ 39us) server_negotiation.cc:407] Connection header received
0924 11:39:46.442746 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:46.442770 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:46.443187 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:46.443614 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:46.445574 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:46.446871 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:46.448958 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:46.450076 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:46.450306 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:46.455070 (+ 4764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:46.455079 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:46.455093 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:39:46.455141 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:46.458397 (+ 3256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:46.459298 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:46.459306 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:46.459314 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:46.459555 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:46.459975 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:46.459981 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:46.459985 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:46.460741 (+ 756us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:46.461104 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:46.461605 (+ 501us) server_negotiation.cc:299] Negotiation successful
0924 11:39:46.462054 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":358,"threads_started":1}
I20260924 11:39:46.465314 29652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:46.449082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49602 (local address 127.6.212.62:43257)
0924 11:39:46.449760 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:46.449767 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:46.450548 (+ 781us) server_negotiation.cc:407] Connection header received
0924 11:39:46.450612 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:46.450615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:46.450680 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:46.450761 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:46.452640 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:46.453389 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:46.454826 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:46.455519 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:46.455652 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:46.459689 (+ 4037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:46.459694 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:46.459698 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:39:46.459732 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:46.462545 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:46.463255 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:46.463262 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:46.463265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:46.463467 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:46.463841 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:46.463849 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:46.463852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:46.464133 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:46.464287 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:46.464792 (+ 505us) server_negotiation.cc:299] Negotiation successful
0924 11:39:46.465013 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":420,"threads_started":1}
I20260924 11:39:46.465617 29653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:46.449140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:43257 (local address 127.0.0.1:49602)
0924 11:39:46.450225 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:46.450266 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 11:39:46.450417 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:39:46.450888 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:39:46.450901 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:39:46.451874 (+ 973us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:39:46.452295 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:46.452314 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:46.453571 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:46.453578 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:46.454675 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:46.454688 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:46.455731 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:46.455736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:46.456033 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:46.456859 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:39:46.456889 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:39:46.459493 (+ 2604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:39:46.462680 (+ 3187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:39:46.462695 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:39:46.462705 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:39:46.463127 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:39:46.463605 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:39:46.463608 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:39:46.463610 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:39:46.463726 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:39:46.464260 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:39:46.464267 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:39:46.464456 (+ 189us) client_negotiation.cc:769] Sending connection context
0924 11:39:46.464659 (+ 203us) client_negotiation.cc:240] Negotiation successful
0924 11:39:46.464987 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":841,"thread_start_us":672,"threads_started":1}
I20260924 11:39:46.467095 29648 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.212.61" port: 45173 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:39:46 dist-test-slave-cq7c krb5kdc[29420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249986, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.60@KRBTEST.COM for kudu/127.6.212.61@KRBTEST.COM
Sep 24 11:39:46 dist-test-slave-cq7c krb5kdc[29420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:39:46.482887 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:46.467893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56688 (local address 127.6.212.61:45173)
0924 11:39:46.468048 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:46.468053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:46.468068 (+ 15us) server_negotiation.cc:407] Connection header received
0924 11:39:46.468151 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:46.468155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:46.468209 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:46.468283 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:46.469194 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:46.470244 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:46.472229 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:46.472972 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:46.473083 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:46.477234 (+ 4151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:46.477243 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:46.477248 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:39:46.477291 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:46.480050 (+ 2759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:46.480831 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:46.480834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:46.480837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:46.480976 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:46.481383 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:46.481388 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:46.481392 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:46.481750 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:46.481918 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:46.482352 (+ 434us) server_negotiation.cc:299] Negotiation successful
0924 11:39:46.482619 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 11:39:46.482872 29653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:46.467721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:45173 (local address 127.0.0.1:56688)
0924 11:39:46.467913 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:46.467932 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 11:39:46.468039 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:39:46.468349 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:39:46.468355 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:39:46.468837 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:39:46.469037 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:46.469053 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:46.470390 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:46.470405 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:46.472077 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:46.472093 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:46.473135 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:46.473143 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:46.473385 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:46.474205 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:39:46.474234 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:39:46.476965 (+ 2731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:39:46.480187 (+ 3222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:39:46.480192 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:39:46.480196 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:39:46.480705 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:39:46.481082 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:39:46.481087 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:39:46.481090 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:39:46.481267 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:39:46.481891 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:39:46.481895 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:39:46.482073 (+ 178us) client_negotiation.cc:769] Sending connection context
0924 11:39:46.482227 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 11:39:46.482521 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 11:39:46.483951 29648 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.212.60" port: 33835 } has no permanent_uuid. Determining permanent_uuid...
I20260924 11:39:46.484911 29652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:46.466776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49616 (local address 127.6.212.62:43257)
0924 11:39:46.466950 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:46.466956 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:46.466985 (+ 29us) server_negotiation.cc:407] Connection header received
0924 11:39:46.467113 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:46.467119 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:46.467229 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:46.467336 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:46.469085 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:46.470179 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:46.472229 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:46.472853 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:46.473023 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:46.477969 (+ 4946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:46.477977 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:46.477984 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:39:46.478034 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:46.481079 (+ 3045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:46.481898 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:46.481903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:46.481906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:46.482072 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:46.482519 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:46.482524 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:46.482527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:46.482870 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:46.483153 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:46.483892 (+ 739us) server_negotiation.cc:299] Negotiation successful
0924 11:39:46.484652 (+ 760us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Sep 24 11:39:46 dist-test-slave-cq7c krb5kdc[29420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249986, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.60@KRBTEST.COM for kudu/127.6.212.60@KRBTEST.COM
Sep 24 11:39:46 dist-test-slave-cq7c krb5kdc[29420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.61@KRBTEST.COM
I20260924 11:39:46.497036 29500 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.212.60" port: 33835 } attempt: 4
I20260924 11:39:46.498159 29651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:46.484912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33578 (local address 127.6.212.60:33835)
0924 11:39:46.485107 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:46.485115 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:46.485316 (+ 201us) server_negotiation.cc:407] Connection header received
0924 11:39:46.485493 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:46.485499 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:46.485587 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:46.485698 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:46.486431 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:46.487508 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:46.488659 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:46.489062 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:46.489202 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:46.492643 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:46.492648 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:46.492651 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:39:46.492679 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:46.495207 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:46.495841 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:46.495849 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:46.495853 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:46.496045 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:46.496477 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:46.496482 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:46.496486 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:46.496848 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:46.497015 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:46.497532 (+ 517us) server_negotiation.cc:299] Negotiation successful
0924 11:39:46.497763 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 11:39:46.500665 29653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:46.484912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:33835 (local address 127.0.0.1:33578)
0924 11:39:46.485217 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:46.485233 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 11:39:46.485344 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:39:46.485731 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:39:46.485734 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:39:46.486159 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:39:46.486292 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:46.486299 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:46.487625 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:46.487632 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:46.488520 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:46.488532 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:46.489220 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:46.489226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:46.489431 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:46.490031 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:39:46.490054 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:39:46.492469 (+ 2415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:39:46.495330 (+ 2861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:39:46.495338 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:39:46.495342 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:39:46.495731 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:39:46.496186 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:39:46.496190 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:39:46.496193 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:39:46.496339 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:39:46.497021 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:39:46.497030 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:39:46.497217 (+ 187us) client_negotiation.cc:769] Sending connection context
0924 11:39:46.497434 (+ 217us) client_negotiation.cc:240] Negotiation successful
0924 11:39:46.498523 (+ 1089us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":695}
I20260924 11:39:46.503921 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:46.486820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56702 (local address 127.6.212.61:45173)
0924 11:39:46.486995 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:46.487000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:46.487018 (+ 18us) server_negotiation.cc:407] Connection header received
0924 11:39:46.487110 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:46.487114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:46.487181 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:46.487273 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:46.488584 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:46.489308 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:46.490736 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:46.491243 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:46.491413 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:46.496134 (+ 4721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:46.496141 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:46.496148 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:39:46.496189 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:46.500719 (+ 4530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:46.501606 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:46.501609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:46.501611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:46.501735 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:46.502191 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:46.502198 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:46.502205 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:46.502547 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:46.502854 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:46.503161 (+ 307us) server_negotiation.cc:299] Negotiation successful
0924 11:39:46.503526 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
Sep 24 11:39:46 dist-test-slave-cq7c krb5kdc[29420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249985, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for kudu/127.6.212.60@KRBTEST.COM
I20260924 11:39:46.507465 29648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d275d51b647043f687b57837ff3220df: Bootstrap starting.
I20260924 11:39:46.509405 6992 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:39:46.511889 29648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d275d51b647043f687b57837ff3220df: Neither blocks nor log segments found. Creating new log.
I20260924 11:39:46.513109 29648 log.cc:824] T 00000000000000000000000000000000 P d275d51b647043f687b57837ff3220df: Log is configured to *not* fsync() on all Append() calls
I20260924 11:39:46.515293 29651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:46.497879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33590 (local address 127.6.212.60:33835)
0924 11:39:46.499205 (+ 1326us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:46.499215 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:46.499243 (+ 28us) server_negotiation.cc:407] Connection header received
0924 11:39:46.499317 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:46.499325 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:46.499460 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:46.499598 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:46.500565 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:46.501759 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:46.503725 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:46.504428 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:46.505146 (+ 718us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:46.508477 (+ 3331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:46.508483 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:46.508487 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:39:46.508520 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:46.512112 (+ 3592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:46.513005 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:46.513010 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:46.513013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:46.513179 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:46.513744 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:46.513752 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:46.513758 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:46.514127 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:46.514290 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:46.514706 (+ 416us) server_negotiation.cc:299] Negotiation successful
0924 11:39:46.514909 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":848,"thread_start_us":2629,"threads_started":1}
I20260924 11:39:46.515377 29504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:46.497728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:33835 (local address 127.0.0.1:33590)
0924 11:39:46.497942 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:46.497963 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 11:39:46.498075 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:39:46.499640 (+ 1565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:39:46.499645 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:39:46.500157 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:39:46.500385 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:46.500398 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:46.501999 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:46.502004 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:46.503420 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:46.503436 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:46.504604 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:46.504610 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:46.504819 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:46.505454 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:39:46.505476 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:39:46.508286 (+ 2810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:39:46.512258 (+ 3972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:39:46.512265 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:39:46.512269 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:39:46.512861 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:39:46.513329 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:39:46.513338 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:39:46.513341 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:39:46.513554 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:39:46.514255 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:39:46.514260 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:39:46.514433 (+ 173us) client_negotiation.cc:769] Sending connection context
0924 11:39:46.514595 (+ 162us) client_negotiation.cc:240] Negotiation successful
0924 11:39:46.514899 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 11:39:46.512355 29475 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "653450b0060b4696be946b14f122d9dd"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49616
I20260924 11:39:46.517531 29648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d275d51b647043f687b57837ff3220df: No bootstrap required, opened a new log
I20260924 11:39:46.521476 29500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 653450b0060b4696be946b14f122d9dd: Bootstrap starting.
I20260924 11:39:46.524518 29500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 653450b0060b4696be946b14f122d9dd: Neither blocks nor log segments found. Creating new log.
I20260924 11:39:46.525784 29500 log.cc:824] T 00000000000000000000000000000000 P 653450b0060b4696be946b14f122d9dd: Log is configured to *not* fsync() on all Append() calls
I20260924 11:39:46.528229 29648 raft_consensus.cc:348] T 00000000000000000000000000000000 P d275d51b647043f687b57837ff3220df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653450b0060b4696be946b14f122d9dd" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43257 } } peers { permanent_uuid: "d4b83c0ee5094dfe941227c6cedaf071" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 45173 } } peers { permanent_uuid: "d275d51b647043f687b57837ff3220df" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 33835 } }
I20260924 11:39:46.528728 29500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 653450b0060b4696be946b14f122d9dd: No bootstrap required, opened a new log
I20260924 11:39:46.528678 29648 raft_consensus.cc:729] T 00000000000000000000000000000000 P d275d51b647043f687b57837ff3220df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d275d51b647043f687b57837ff3220df, State: Initialized, Role: FOLLOWER
I20260924 11:39:46.529217 29648 consensus_queue.cc:260] T 00000000000000000000000000000000 P d275d51b647043f687b57837ff3220df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653450b0060b4696be946b14f122d9dd" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43257 } } peers { permanent_uuid: "d4b83c0ee5094dfe941227c6cedaf071" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 45173 } } peers { permanent_uuid: "d275d51b647043f687b57837ff3220df" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 33835 } }
I20260924 11:39:46.530201 29657 sys_catalog.cc:455] T 00000000000000000000000000000000 P d275d51b647043f687b57837ff3220df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653450b0060b4696be946b14f122d9dd" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43257 } } peers { permanent_uuid: "d4b83c0ee5094dfe941227c6cedaf071" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 45173 } } peers { permanent_uuid: "d275d51b647043f687b57837ff3220df" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 33835 } } }
I20260924 11:39:46.530632 29657 sys_catalog.cc:458] T 00000000000000000000000000000000 P d275d51b647043f687b57837ff3220df [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:39:46.531596 29648 sys_catalog.cc:564] T 00000000000000000000000000000000 P d275d51b647043f687b57837ff3220df [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:39:46.535586 29500 raft_consensus.cc:348] T 00000000000000000000000000000000 P 653450b0060b4696be946b14f122d9dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653450b0060b4696be946b14f122d9dd" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43257 } } peers { permanent_uuid: "d4b83c0ee5094dfe941227c6cedaf071" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 45173 } } peers { permanent_uuid: "d275d51b647043f687b57837ff3220df" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 33835 } }
I20260924 11:39:46.535974 29500 raft_consensus.cc:729] T 00000000000000000000000000000000 P 653450b0060b4696be946b14f122d9dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 653450b0060b4696be946b14f122d9dd, State: Initialized, Role: FOLLOWER
I20260924 11:39:46.536475 29500 consensus_queue.cc:260] T 00000000000000000000000000000000 P 653450b0060b4696be946b14f122d9dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653450b0060b4696be946b14f122d9dd" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43257 } } peers { permanent_uuid: "d4b83c0ee5094dfe941227c6cedaf071" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 45173 } } peers { permanent_uuid: "d275d51b647043f687b57837ff3220df" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 33835 } }
I20260924 11:39:46.538133 29669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 653450b0060b4696be946b14f122d9dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653450b0060b4696be946b14f122d9dd" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43257 } } peers { permanent_uuid: "d4b83c0ee5094dfe941227c6cedaf071" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 45173 } } peers { permanent_uuid: "d275d51b647043f687b57837ff3220df" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 33835 } } }
I20260924 11:39:46.538636 29669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 653450b0060b4696be946b14f122d9dd [sys.catalog]: This master's current role is: FOLLOWER
W20260924 11:39:46.541795 29668 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d275d51b647043f687b57837ff3220df: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 11:39:46.541920 29668 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 11:39:46.539688 29581 master_runner.cc:186] Error getting master registration for 127.6.212.62:43257: OK, instance_id { permanent_uuid: "653450b0060b4696be946b14f122d9dd" instance_seqno: 1790249985348677 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 11:39:46.547801 29581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 11:39:46.549048 29500 sys_catalog.cc:564] T 00000000000000000000000000000000 P 653450b0060b4696be946b14f122d9dd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:39:46.549158 29475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 653450b0060b4696be946b14f122d9dd [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 11:39:46.549366 29475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 653450b0060b4696be946b14f122d9dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:39:46.553579 29475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 653450b0060b4696be946b14f122d9dd [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653450b0060b4696be946b14f122d9dd" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43257 } } peers { permanent_uuid: "d4b83c0ee5094dfe941227c6cedaf071" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 45173 } } peers { permanent_uuid: "d275d51b647043f687b57837ff3220df" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 33835 } }
I20260924 11:39:46.556213 29475 leader_election.cc:288] T 00000000000000000000000000000000 P 653450b0060b4696be946b14f122d9dd [CANDIDATE]: Term 1 election: Requested vote from peers d4b83c0ee5094dfe941227c6cedaf071 (127.6.212.61:45173), d275d51b647043f687b57837ff3220df (127.6.212.60:33835)
I20260924 11:39:46.556519 29623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "653450b0060b4696be946b14f122d9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d275d51b647043f687b57837ff3220df"
I20260924 11:39:46.557013 29623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d275d51b647043f687b57837ff3220df [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:39:46.560359 29623 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d275d51b647043f687b57837ff3220df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 653450b0060b4696be946b14f122d9dd in term 1.
I20260924 11:39:46.561637 29451 leader_election.cc:302] T 00000000000000000000000000000000 P 653450b0060b4696be946b14f122d9dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 653450b0060b4696be946b14f122d9dd, d275d51b647043f687b57837ff3220df; no voters:
I20260924 11:39:46.562208 29669 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 653450b0060b4696be946b14f122d9dd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:39:46.571657 29669 raft_consensus.cc:686] T 00000000000000000000000000000000 P 653450b0060b4696be946b14f122d9dd [term 1 LEADER]: Becoming Leader. State: Replica: 653450b0060b4696be946b14f122d9dd, State: Running, Role: LEADER
I20260924 11:39:46.572422 29669 consensus_queue.cc:237] T 00000000000000000000000000000000 P 653450b0060b4696be946b14f122d9dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653450b0060b4696be946b14f122d9dd" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43257 } } peers { permanent_uuid: "d4b83c0ee5094dfe941227c6cedaf071" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 45173 } } peers { permanent_uuid: "d275d51b647043f687b57837ff3220df" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 33835 } }
W20260924 11:39:46.586829 29454 tablet.cc:2369] T 00000000000000000000000000000000 P 653450b0060b4696be946b14f122d9dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:39:46.596618 29678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 653450b0060b4696be946b14f122d9dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 653450b0060b4696be946b14f122d9dd. Latest consensus state: current_term: 1 leader_uuid: "653450b0060b4696be946b14f122d9dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653450b0060b4696be946b14f122d9dd" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43257 } } peers { permanent_uuid: "d4b83c0ee5094dfe941227c6cedaf071" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 45173 } } peers { permanent_uuid: "d275d51b647043f687b57837ff3220df" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 33835 } } }
I20260924 11:39:46.597095 29678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 653450b0060b4696be946b14f122d9dd [sys.catalog]: This master's current role is: LEADER
I20260924 11:39:46.606055 29689 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 11:39:46.613674 29574 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.212.60" port: 33835 } attempt: 5
Sep 24 11:39:46 dist-test-slave-cq7c krb5kdc[29420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249985, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.61@KRBTEST.COM for kudu/127.6.212.60@KRBTEST.COM
I20260924 11:39:46.632433 29577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:46.614244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:33835 (local address 127.0.0.1:33610)
0924 11:39:46.614626 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:46.614646 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 11:39:46.614750 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:39:46.615854 (+ 1104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:39:46.615859 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:39:46.616491 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:39:46.616713 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:46.616726 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:46.618458 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:46.618464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:46.620170 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:46.620184 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:46.621404 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:46.621412 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:46.621693 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:46.622280 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:39:46.622384 (+ 104us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:39:46.625179 (+ 2795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:39:46.629557 (+ 4378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:39:46.629565 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:39:46.629569 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:39:46.630082 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:39:46.630531 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:39:46.630538 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:39:46.630545 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:39:46.630733 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:39:46.631438 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:39:46.631444 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:39:46.631600 (+ 156us) client_negotiation.cc:769] Sending connection context
0924 11:39:46.631751 (+ 151us) client_negotiation.cc:240] Negotiation successful
0924 11:39:46.632077 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"spinlock_wait_cycles":27008}
I20260924 11:39:46.633183 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:46.583280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56716 (local address 127.6.212.61:45173)
0924 11:39:46.583547 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:46.583556 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:46.591439 (+ 7883us) server_negotiation.cc:407] Connection header received
0924 11:39:46.592207 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:46.592225 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:46.592339 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:46.592434 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:46.607490 (+ 15056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:46.608616 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:46.611489 (+ 2873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:46.612146 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:46.612248 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:46.616698 (+ 4450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:46.616703 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:46.616708 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:39:46.616757 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:46.620543 (+ 3786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:46.622266 (+ 1723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:46.622272 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:46.622275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:46.622442 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:46.624399 (+ 1957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:46.624404 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:46.624409 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:46.624773 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:46.625014 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:46.632567 (+ 7553us) server_negotiation.cc:299] Negotiation successful
0924 11:39:46.632802 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260924 11:39:46.635381 29654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:46.615000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33610 (local address 127.6.212.60:33835)
0924 11:39:46.615268 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:46.615274 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:46.615306 (+ 32us) server_negotiation.cc:407] Connection header received
0924 11:39:46.615371 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:46.615376 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:46.615454 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:46.615562 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:46.616958 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:46.618294 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:46.620342 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:46.621051 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:46.621180 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:46.625404 (+ 4224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:46.625411 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:46.625416 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:39:46.625454 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:46.629388 (+ 3934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:46.630214 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:46.630219 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:46.630223 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:46.630384 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:46.630877 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:46.630882 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:46.630886 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:46.631269 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:46.634593 (+ 3324us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:46.634709 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 11:39:46.634911 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260924 11:39:46.638852 29651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:46.589639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33606 (local address 127.6.212.60:33835)
0924 11:39:46.589899 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:46.589908 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:46.594931 (+ 5023us) server_negotiation.cc:407] Connection header received
0924 11:39:46.598779 (+ 3848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:46.598788 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:46.598915 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:46.599006 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:46.605945 (+ 6939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:46.607119 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:46.612531 (+ 5412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:46.613388 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:46.613498 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:46.629918 (+ 16420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:46.629924 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:46.629929 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:39:46.629968 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:46.634384 (+ 4416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:46.635128 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:46.635135 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:46.635144 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:46.635280 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:46.636510 (+ 1230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:46.636518 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:46.636527 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:46.636859 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:46.637106 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:46.637614 (+ 508us) server_negotiation.cc:299] Negotiation successful
0924 11:39:46.637851 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260924 11:39:46.643297 29574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4b83c0ee5094dfe941227c6cedaf071: Bootstrap starting.
I20260924 11:39:46.649250 29689 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:39:46.650825 29574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4b83c0ee5094dfe941227c6cedaf071: Neither blocks nor log segments found. Creating new log.
I20260924 11:39:46.652868 29574 log.cc:824] T 00000000000000000000000000000000 P d4b83c0ee5094dfe941227c6cedaf071: Log is configured to *not* fsync() on all Append() calls
W20260924 11:39:46.661446 29602 tablet.cc:2369] T 00000000000000000000000000000000 P d275d51b647043f687b57837ff3220df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:39:46.661681 29652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:46.592703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49632 (local address 127.6.212.62:43257)
0924 11:39:46.592904 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:46.592910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:46.598456 (+ 5546us) server_negotiation.cc:407] Connection header received
0924 11:39:46.601969 (+ 3513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:46.601979 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:46.602083 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:46.602191 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:46.609089 (+ 6898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:46.610214 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:46.626978 (+ 16764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:46.638446 (+ 11468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:46.638583 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:46.642467 (+ 3884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:46.642474 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:46.642479 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:39:46.642517 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:46.647983 (+ 5466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:46.656373 (+ 8390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:46.656380 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:46.656384 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:46.656590 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:46.657196 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:46.657202 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:46.657206 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:46.659808 (+ 2602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:46.660305 (+ 497us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:46.660568 (+ 263us) server_negotiation.cc:299] Negotiation successful
0924 11:39:46.660998 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 11:39:46.663417 29694 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 11:39:46.671814 29574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4b83c0ee5094dfe941227c6cedaf071: No bootstrap required, opened a new log
I20260924 11:39:46.685851 29623 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d275d51b647043f687b57837ff3220df [term 1 FOLLOWER]: Refusing update from remote peer 653450b0060b4696be946b14f122d9dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:39:46.689750 29678 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 653450b0060b4696be946b14f122d9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d275d51b647043f687b57837ff3220df" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 33835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:39:46.696614 29574 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4b83c0ee5094dfe941227c6cedaf071 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653450b0060b4696be946b14f122d9dd" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43257 } } peers { permanent_uuid: "d4b83c0ee5094dfe941227c6cedaf071" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 45173 } } peers { permanent_uuid: "d275d51b647043f687b57837ff3220df" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 33835 } }
I20260924 11:39:46.697296 29574 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4b83c0ee5094dfe941227c6cedaf071 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4b83c0ee5094dfe941227c6cedaf071, State: Initialized, Role: FOLLOWER
I20260924 11:39:46.698141 29574 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4b83c0ee5094dfe941227c6cedaf071 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653450b0060b4696be946b14f122d9dd" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43257 } } peers { permanent_uuid: "d4b83c0ee5094dfe941227c6cedaf071" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 45173 } } peers { permanent_uuid: "d275d51b647043f687b57837ff3220df" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 33835 } }
I20260924 11:39:46.671855 29433 master_runner.cc:186] Error getting master registration for 127.6.212.61:45173: OK, instance_id { permanent_uuid: "d4b83c0ee5094dfe941227c6cedaf071" instance_seqno: 1790249985869059 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 11:39:46.706884 29433 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 11:39:46.709450 29549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "653450b0060b4696be946b14f122d9dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d4b83c0ee5094dfe941227c6cedaf071"
I20260924 11:39:46.709828 29548 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4b83c0ee5094dfe941227c6cedaf071 [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:39:46.710158 29574 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4b83c0ee5094dfe941227c6cedaf071 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:39:46.714181 29548 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d4b83c0ee5094dfe941227c6cedaf071 [term 1 FOLLOWER]: Refusing update from remote peer 653450b0060b4696be946b14f122d9dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:39:46.715322 29678 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 653450b0060b4696be946b14f122d9dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4b83c0ee5094dfe941227c6cedaf071" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 45173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:39:46.717876 29549 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d4b83c0ee5094dfe941227c6cedaf071 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 653450b0060b4696be946b14f122d9dd in term 1.
I20260924 11:39:46.721798 29696 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4b83c0ee5094dfe941227c6cedaf071 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653450b0060b4696be946b14f122d9dd" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43257 } } peers { permanent_uuid: "d4b83c0ee5094dfe941227c6cedaf071" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 45173 } } peers { permanent_uuid: "d275d51b647043f687b57837ff3220df" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 33835 } } }
I20260924 11:39:46.722254 29696 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4b83c0ee5094dfe941227c6cedaf071 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:39:46.723160 29657 sys_catalog.cc:455] T 00000000000000000000000000000000 P d275d51b647043f687b57837ff3220df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 653450b0060b4696be946b14f122d9dd. Latest consensus state: current_term: 1 leader_uuid: "653450b0060b4696be946b14f122d9dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653450b0060b4696be946b14f122d9dd" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43257 } } peers { permanent_uuid: "d4b83c0ee5094dfe941227c6cedaf071" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 45173 } } peers { permanent_uuid: "d275d51b647043f687b57837ff3220df" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 33835 } } }
I20260924 11:39:46.723541 29657 sys_catalog.cc:458] T 00000000000000000000000000000000 P d275d51b647043f687b57837ff3220df [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:39:46.724138 29693 mvcc.cc:205] Tried to move back new op lower bound from 7332863945440280576 to 7332863945008312320. Current Snapshot: MvccSnapshot[applied={T|T < 7332863945440280576}]
I20260924 11:39:46.725822 29678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 653450b0060b4696be946b14f122d9dd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "653450b0060b4696be946b14f122d9dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653450b0060b4696be946b14f122d9dd" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43257 } } peers { permanent_uuid: "d4b83c0ee5094dfe941227c6cedaf071" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 45173 } } peers { permanent_uuid: "d275d51b647043f687b57837ff3220df" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 33835 } } }
I20260924 11:39:46.726176 29678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 653450b0060b4696be946b14f122d9dd [sys.catalog]: This master's current role is: LEADER
I20260924 11:39:46.728282 29689 catalog_manager.cc:1347] Generated new cluster ID: 0497e154179b44698e36ca66026d5f8d
I20260924 11:39:46.728468 29689 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:39:46.749321 29657 sys_catalog.cc:455] T 00000000000000000000000000000000 P d275d51b647043f687b57837ff3220df [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "653450b0060b4696be946b14f122d9dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653450b0060b4696be946b14f122d9dd" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43257 } } peers { permanent_uuid: "d4b83c0ee5094dfe941227c6cedaf071" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 45173 } } peers { permanent_uuid: "d275d51b647043f687b57837ff3220df" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 33835 } } }
I20260924 11:39:46.749627 29657 sys_catalog.cc:458] T 00000000000000000000000000000000 P d275d51b647043f687b57837ff3220df [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:39:46.754031 29696 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4b83c0ee5094dfe941227c6cedaf071 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 653450b0060b4696be946b14f122d9dd. Latest consensus state: current_term: 1 leader_uuid: "653450b0060b4696be946b14f122d9dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653450b0060b4696be946b14f122d9dd" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43257 } } peers { permanent_uuid: "d4b83c0ee5094dfe941227c6cedaf071" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 45173 } } peers { permanent_uuid: "d275d51b647043f687b57837ff3220df" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 33835 } } }
I20260924 11:39:46.754318 29696 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4b83c0ee5094dfe941227c6cedaf071 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:39:46.755867 29669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 653450b0060b4696be946b14f122d9dd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "653450b0060b4696be946b14f122d9dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653450b0060b4696be946b14f122d9dd" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43257 } } peers { permanent_uuid: "d4b83c0ee5094dfe941227c6cedaf071" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 45173 } } peers { permanent_uuid: "d275d51b647043f687b57837ff3220df" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 33835 } } }
I20260924 11:39:46.756217 29669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 653450b0060b4696be946b14f122d9dd [sys.catalog]: This master's current role is: LEADER
I20260924 11:39:46.771497 29696 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4b83c0ee5094dfe941227c6cedaf071 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "653450b0060b4696be946b14f122d9dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "653450b0060b4696be946b14f122d9dd" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 43257 } } peers { permanent_uuid: "d4b83c0ee5094dfe941227c6cedaf071" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 45173 } } peers { permanent_uuid: "d275d51b647043f687b57837ff3220df" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 33835 } } }
I20260924 11:39:46.771766 29696 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4b83c0ee5094dfe941227c6cedaf071 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:39:46.781706 29689 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:39:46.783701 29689 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:39:46.804141 29689 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 653450b0060b4696be946b14f122d9dd: Generated new TSK 0
I20260924 11:39:46.805225 29689 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:39:46.829409 29715 catalog_manager.cc:1259] Loaded cluster ID: 0497e154179b44698e36ca66026d5f8d
I20260924 11:39:46.829509 29715 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d4b83c0ee5094dfe941227c6cedaf071: loading cluster ID for follower catalog manager: success
I20260924 11:39:46.840344 29715 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d4b83c0ee5094dfe941227c6cedaf071: acquiring CA information for follower catalog manager: success
I20260924 11:39:46.842933 29715 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d4b83c0ee5094dfe941227c6cedaf071: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 11:39:46.971948 29578 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:39:46.952794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56730 (local address 127.6.212.61:45173)
0924 11:39:46.953000 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:46.953011 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:46.969936 (+ 16925us) server_negotiation.cc:407] Connection header received
0924 11:39:46.970283 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:46.970289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:46.970389 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:46.970465 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:46.971710 (+ 1245us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56730: BlockingRecv error: recv got EOF from 127.0.0.1:56730 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260924 11:39:46.974258 29651 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:39:46.959066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33622 (local address 127.6.212.60:33835)
0924 11:39:46.959288 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:46.959295 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:46.972559 (+ 13264us) server_negotiation.cc:407] Connection header received
0924 11:39:46.972813 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:46.972821 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:46.972906 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:46.972971 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:46.973941 (+ 970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33622: BlockingRecv error: recv got EOF from 127.0.0.1:33622 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
W20260924 11:39:46.975011 29652 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:39:46.969774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49646 (local address 127.6.212.62:43257)
0924 11:39:46.969973 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:46.969979 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:46.972989 (+ 3010us) server_negotiation.cc:407] Connection header received
0924 11:39:46.973580 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:46.973585 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:46.973667 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:46.973741 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:46.974772 (+ 1031us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49646: BlockingRecv error: recv got EOF from 127.0.0.1:49646 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 11:39:47.031390 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:47.012656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56744 (local address 127.6.212.61:45173)
0924 11:39:47.012847 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:47.012854 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:47.022960 (+ 10106us) server_negotiation.cc:407] Connection header received
0924 11:39:47.023573 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:47.023581 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:47.023799 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:47.023969 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:39:47.024712 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:47.025941 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:47.028252 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:47.028906 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:47.029001 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:47.029778 (+ 777us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:39:47.030495 (+ 717us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:39:47.030619 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:47.030846 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 11:39:47.030957 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260924 11:39:47.074997 29651 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:39:47.065401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33626 (local address 127.6.212.60:33835)
0924 11:39:47.065636 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:47.065643 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:47.072985 (+ 7342us) server_negotiation.cc:407] Connection header received
0924 11:39:47.073515 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:47.073522 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:47.073654 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:47.073740 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:47.074761 (+ 1021us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33626: BlockingRecv error: recv got EOF from 127.0.0.1:33626 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 11:39:47.123878 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:47.105955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56754 (local address 127.6.212.61:45173)
0924 11:39:47.106186 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:47.106194 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:47.117067 (+ 10873us) server_negotiation.cc:407] Connection header received
0924 11:39:47.117379 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:47.117386 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:47.117501 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:47.117568 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:39:47.118312 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:47.119492 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:47.121538 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:47.122205 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:47.122303 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:47.122934 (+ 631us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:39:47.123119 (+ 185us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:39:47.123187 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:47.123452 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 11:39:47.123541 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260924 11:39:47.170217 29651 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:39:47.158754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33636 (local address 127.6.212.60:33835)
0924 11:39:47.158962 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:47.158968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:47.168374 (+ 9406us) server_negotiation.cc:407] Connection header received
0924 11:39:47.168690 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:47.168696 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:47.168806 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:47.168889 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:47.169971 (+ 1082us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33636: BlockingRecv error: recv got EOF from 127.0.0.1:33636 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 11:39:47.198449 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:47.184038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56756 (local address 127.6.212.61:45173)
0924 11:39:47.184274 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:47.184282 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:47.191594 (+ 7312us) server_negotiation.cc:407] Connection header received
0924 11:39:47.191847 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:47.191853 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:47.191953 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:47.192074 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:39:47.192555 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:47.193765 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:47.195915 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:47.196901 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:47.197036 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:47.197533 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:39:47.197712 (+ 179us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:39:47.197814 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:47.198000 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 11:39:47.198087 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260924 11:39:47.209697 29651 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:39:47.206708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33640 (local address 127.6.212.60:33835)
0924 11:39:47.206898 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:47.206904 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:47.207954 (+ 1050us) server_negotiation.cc:407] Connection header received
0924 11:39:47.208254 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:47.208262 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:47.208371 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:47.208463 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:47.209444 (+ 981us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33640: BlockingRecv error: recv got EOF from 127.0.0.1:33640 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 11:39:47.225355 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:47.218207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56758 (local address 127.6.212.61:45173)
0924 11:39:47.218462 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:47.218469 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:47.219005 (+ 536us) server_negotiation.cc:407] Connection header received
0924 11:39:47.219238 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:47.219244 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:47.219335 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:47.219456 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:39:47.219954 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:47.221116 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:47.223142 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:47.223930 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:47.224068 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:47.224562 (+ 494us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:39:47.224712 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:39:47.224807 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:47.224985 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 11:39:47.225073 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
W20260924 11:39:47.240460 29651 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:39:47.233317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33654 (local address 127.6.212.60:33835)
0924 11:39:47.233588 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:47.233596 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:47.238444 (+ 4848us) server_negotiation.cc:407] Connection header received
0924 11:39:47.238673 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:47.238680 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:47.238800 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:47.238923 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:47.240011 (+ 1088us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33654: BlockingRecv error: recv got EOF from 127.0.0.1:33654 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
I20260924 11:39:47.259436 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:47.251861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56770 (local address 127.6.212.61:45173)
0924 11:39:47.252065 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:47.252073 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:47.252566 (+ 493us) server_negotiation.cc:407] Connection header received
0924 11:39:47.252913 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:47.252919 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:47.253018 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:47.253091 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:39:47.253775 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:47.255011 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:47.257095 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:47.257830 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:47.257964 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:47.258465 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:39:47.258633 (+ 168us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:39:47.258735 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:47.258922 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 11:39:47.259008 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260924 11:39:47.278263 29651 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:39:47.267300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33670 (local address 127.6.212.60:33835)
0924 11:39:47.267516 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:47.267523 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:47.276245 (+ 8722us) server_negotiation.cc:407] Connection header received
0924 11:39:47.276610 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:47.276617 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:47.276724 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:47.276808 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:47.278033 (+ 1225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33670: BlockingRecv error: recv got EOF from 127.0.0.1:33670 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
I20260924 11:39:47.309237 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:47.301721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56776 (local address 127.6.212.61:45173)
0924 11:39:47.301938 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:47.301945 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:47.302499 (+ 554us) server_negotiation.cc:407] Connection header received
0924 11:39:47.302750 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:47.302757 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:47.302849 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:47.302970 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:39:47.303502 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:47.304837 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:47.306895 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:47.307665 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:47.307825 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:47.308313 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:39:47.308464 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:39:47.308563 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:47.308733 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 11:39:47.308840 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260924 11:39:47.319993 29651 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:39:47.317503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33672 (local address 127.6.212.60:33835)
0924 11:39:47.317716 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:47.317723 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:47.318303 (+ 580us) server_negotiation.cc:407] Connection header received
0924 11:39:47.318529 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:47.318535 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:47.318629 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:47.318726 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:47.319755 (+ 1029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33672: BlockingRecv error: recv got EOF from 127.0.0.1:33672 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 11:39:47.364409 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:47.356994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56786 (local address 127.6.212.61:45173)
0924 11:39:47.357243 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:47.357251 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:47.357626 (+ 375us) server_negotiation.cc:407] Connection header received
0924 11:39:47.357834 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:47.357840 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:47.357948 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:47.358046 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:39:47.358551 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:47.359712 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:47.361951 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:47.362705 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:47.362827 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:47.363416 (+ 589us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:39:47.363563 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:39:47.363662 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:47.363826 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 11:39:47.363910 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260924 11:39:47.375597 29651 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:39:47.373361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33682 (local address 127.6.212.60:33835)
0924 11:39:47.373541 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:47.373547 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:47.374092 (+ 545us) server_negotiation.cc:407] Connection header received
0924 11:39:47.374272 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:47.374279 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:47.374380 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:47.374497 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:47.375323 (+ 826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33682: BlockingRecv error: recv got EOF from 127.0.0.1:33682 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 11:39:47.449162 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:47.443019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56790 (local address 127.6.212.61:45173)
0924 11:39:47.443273 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:47.443280 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:47.443633 (+ 353us) server_negotiation.cc:407] Connection header received
0924 11:39:47.443877 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:47.443881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:47.443981 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:47.444069 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:39:47.444569 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:47.445322 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:47.447249 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:47.447777 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:47.447882 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:47.448341 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:39:47.448511 (+ 170us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:39:47.448581 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:47.448734 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 11:39:47.448799 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260924 11:39:47.457368 29651 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:39:47.455255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33698 (local address 127.6.212.60:33835)
0924 11:39:47.455613 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:47.455621 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:47.455974 (+ 353us) server_negotiation.cc:407] Connection header received
0924 11:39:47.456196 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:47.456201 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:47.456272 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:47.456379 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:47.457183 (+ 804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33698: BlockingRecv error: recv got EOF from 127.0.0.1:33698 (error 108)
Metrics: {"server-negotiator.queue_time_us":179}
I20260924 11:39:47.544195 29668 catalog_manager.cc:1259] Loaded cluster ID: 0497e154179b44698e36ca66026d5f8d
I20260924 11:39:47.544332 29668 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d275d51b647043f687b57837ff3220df: loading cluster ID for follower catalog manager: success
I20260924 11:39:47.546861 29668 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d275d51b647043f687b57837ff3220df: acquiring CA information for follower catalog manager: success
I20260924 11:39:47.548723 29668 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d275d51b647043f687b57837ff3220df: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 11:39:47.596072 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:47.590773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56798 (local address 127.6.212.61:45173)
0924 11:39:47.591054 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:47.591060 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:47.591283 (+ 223us) server_negotiation.cc:407] Connection header received
0924 11:39:47.591549 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:47.591553 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:47.591625 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:47.591711 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:39:47.592131 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:47.592941 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:47.594246 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:47.594705 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:47.594806 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:47.595173 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:39:47.595283 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:39:47.595349 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:47.595716 (+ 367us) server_negotiation.cc:299] Negotiation successful
0924 11:39:47.595820 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 11:39:47.608646 29651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:47.602323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33702 (local address 127.6.212.60:33835)
0924 11:39:47.602597 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:47.602606 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:47.602894 (+ 288us) server_negotiation.cc:407] Connection header received
0924 11:39:47.603123 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:47.603134 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:47.603321 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:47.603495 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:39:47.603983 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:47.604694 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:47.606234 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:47.606683 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:47.606837 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:47.607425 (+ 588us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:39:47.608015 (+ 590us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:39:47.608199 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:47.608259 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 11:39:47.608349 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260924 11:39:47.626008 29821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:47.615380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49648 (local address 127.6.212.62:43257)
0924 11:39:47.619833 (+ 4453us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:47.619842 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:47.619871 (+ 29us) server_negotiation.cc:407] Connection header received
0924 11:39:47.620011 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:47.620019 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:47.620264 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:47.620465 (+ 201us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:39:47.621012 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:47.621839 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:47.623282 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:47.623953 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:47.624084 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:47.624652 (+ 568us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:39:47.625097 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:39:47.625199 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:47.625644 (+ 445us) server_negotiation.cc:299] Negotiation successful
0924 11:39:47.625733 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4284,"thread_start_us":4160,"threads_started":1}
I20260924 11:39:47.627086 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:47.615630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56800 (local address 127.6.212.61:45173)
0924 11:39:47.615912 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:47.615920 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:47.620337 (+ 4417us) server_negotiation.cc:407] Connection header received
0924 11:39:47.620672 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:47.620679 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:47.620803 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:47.620872 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:39:47.621881 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:47.623005 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:47.624906 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:47.625477 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:47.625612 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:47.626228 (+ 616us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:39:47.626356 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:39:47.626438 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:47.626676 (+ 238us) server_negotiation.cc:299] Negotiation successful
0924 11:39:47.626782 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260924 11:39:47.628063 29651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:47.617051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33708 (local address 127.6.212.60:33835)
0924 11:39:47.617306 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:47.617315 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:47.620585 (+ 3270us) server_negotiation.cc:407] Connection header received
0924 11:39:47.620872 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:47.620878 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:47.621006 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:47.621102 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:39:47.622033 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:47.623041 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:47.625199 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:47.625864 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:47.625993 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:47.626541 (+ 548us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:39:47.626685 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:39:47.626757 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:47.627691 (+ 934us) server_negotiation.cc:299] Negotiation successful
0924 11:39:47.627774 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 11:39:47.631345 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 29433
I20260924 11:39:47.632477 29495 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:39:47.748193 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 29433
I20260924 11:39:47.771939 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 29507
I20260924 11:39:47.773218 29569 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:39:47.887171 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 29507
I20260924 11:39:47.908156 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 29581
I20260924 11:39:47.909286 29643 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:39:48.017848 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 29581
2026-09-24T11:39:48Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4533 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 11:39:48 dist-test-slave-cq7c krb5kdc[29829](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 11:39:48 dist-test-slave-cq7c krb5kdc[29829](info): set up 2 sockets
Sep 24 11:39:48 dist-test-slave-cq7c krb5kdc[29829](info): commencing operation
krb5kdc: starting...
W20260924 11:39:49.156054 6992 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.004s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/krb5kdc/test-user.keytab.
Sep 24 11:39:49 dist-test-slave-cq7c krb5kdc[29829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T11:39:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T11:39:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.212.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
Entry for principal kudu/127.6.212.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab.
I20260924 11:39:49.447274 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:36299
--webserver_interface=127.6.212.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:33451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:36299,127.6.212.61:44305,127.6.212.60:34435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 11:39:49.795593 29842 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 11:39:49.796044 29842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:39:49.796204 29842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:39:49.796387 29842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:39:49.804980 29842 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:39:49.805135 29842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:39:49.805286 29842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:39:49.805395 29842 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:39:49.805509 29842 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:39:49.817307 29842 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:33451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:36299,127.6.212.61:44305,127.6.212.60:34435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.62.keytab
--principal=kudu/127.6.212.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.62:36299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.212.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:39:49.818876 29842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:39:49.820570 29842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:39:49.834434 29848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:49.834548 29850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:49.835161 29847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:39:49.835489 29842 server_base.cc:1034] running on GCE node
I20260924 11:39:49.836324 29842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:39:49.837744 29842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:39:49.839010 29842 hybrid_clock.cc:648] HybridClock initialized: now 1790249989838943 us; error 44 us; skew 500 ppm
I20260924 11:39:49.839389 29842 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:39:49 dist-test-slave-cq7c krb5kdc[29829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249989, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:39:49.842691 29842 init.cc:377] Logged in from keytab as kudu/127.6.212.62@KRBTEST.COM (short username kudu)
I20260924 11:39:49.844211 29842 webserver.cc:460] Webserver started at http://127.6.212.62:45167/ using document root <none> and password file <none>
I20260924 11:39:49.844882 29842 fs_manager.cc:359] Metadata directory not provided
I20260924 11:39:49.845029 29842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:39:49.845369 29842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:39:49.848443 29842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-0/data/instance:
uuid: "95e2baa7c71c40c785fcf6d604c22eee"
format_stamp: "Formatted at 2026-09-24 11:39:49 on dist-test-slave-cq7c"
server_key: "e878b0e04cf0775b2fdbf772627bc870"
server_key_iv: "b263a389207ed9e160dac989a3f58044"
server_key_version: "encryptionkey@0"
I20260924 11:39:49.849157 29842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-0/wal/instance:
uuid: "95e2baa7c71c40c785fcf6d604c22eee"
format_stamp: "Formatted at 2026-09-24 11:39:49 on dist-test-slave-cq7c"
server_key: "e878b0e04cf0775b2fdbf772627bc870"
server_key_iv: "b263a389207ed9e160dac989a3f58044"
server_key_version: "encryptionkey@0"
I20260924 11:39:49.854295 29842 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 11:39:49.857956 29857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:39:49.859520 29842 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:39:49.859696 29842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-0/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-0/wal
uuid: "95e2baa7c71c40c785fcf6d604c22eee"
format_stamp: "Formatted at 2026-09-24 11:39:49 on dist-test-slave-cq7c"
server_key: "e878b0e04cf0775b2fdbf772627bc870"
server_key_iv: "b263a389207ed9e160dac989a3f58044"
server_key_version: "encryptionkey@0"
I20260924 11:39:49.859932 29842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:39:49.875037 29842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:39:49.879231 29842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:39:49.879567 29842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:39:49.898809 29842 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.62:36299
I20260924 11:39:49.898838 29908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.62:36299 every 8 connection(s)
I20260924 11:39:49.900229 29842 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-0/data/info.pb
I20260924 11:39:49.904122 29909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:39:49.906402 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 29842
I20260924 11:39:49.906611 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-0/wal/instance
I20260924 11:39:49.907051 6992 external_mini_cluster.cc:1428] Setting key c2529aca66da5d7105f1dd584851e25a
I20260924 11:39:49.910682 29909 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36299 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:39:49 dist-test-slave-cq7c krb5kdc[29829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249989, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.212.61@KRBTEST.COM; defaulting to no policy
I20260924 11:39:49.933136 29912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:49.913848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36299 (local address 127.0.0.1:35254)
0924 11:39:49.914961 (+ 1113us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:49.915002 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 11:39:49.916795 (+ 1793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:39:49.917651 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:39:49.917657 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:39:49.918374 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:39:49.918705 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:49.918722 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:49.920374 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:49.920381 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:49.921460 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:49.921471 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:49.922396 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:49.922411 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:49.922590 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:49.923279 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:39:49.923303 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:39:49.925531 (+ 2228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:39:49.928899 (+ 3368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:39:49.928911 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:39:49.928925 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:39:49.929315 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:39:49.929725 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:39:49.929733 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:39:49.929738 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:39:49.929888 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:39:49.930657 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:39:49.930666 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:39:49.930873 (+ 207us) client_negotiation.cc:769] Sending connection context
0924 11:39:49.931123 (+ 250us) client_negotiation.cc:240] Negotiation successful
0924 11:39:49.931722 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":913,"mutex_wait_us":137,"thread_start_us":711,"threads_started":1}
I20260924 11:39:49.933169 29911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:49.913847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35254 (local address 127.6.212.62:36299)
0924 11:39:49.914676 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:49.914695 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:49.915128 (+ 433us) server_negotiation.cc:407] Connection header received
0924 11:39:49.917181 (+ 2053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:49.917187 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:49.917434 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:49.917639 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:49.918928 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:49.920218 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:49.921609 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:49.922161 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:49.922356 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:49.925744 (+ 3388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:49.925752 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:49.925765 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:39:49.925793 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:49.928768 (+ 2975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:49.929464 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:49.929472 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:49.929476 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:49.929619 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:49.929990 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:49.929998 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:49.930001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:49.930521 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:49.930732 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:49.931235 (+ 503us) server_negotiation.cc:299] Negotiation successful
0924 11:39:49.931605 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":571,"thread_start_us":357,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.61@KRBTEST.COM" created.
I20260924 11:39:49.936321 29909 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.212.61" port: 44305 } has no permanent_uuid. Determining permanent_uuid...
I20260924 11:39:49.937626 29912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:49.937104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:44305 (local address 127.0.0.1:51956)
0924 11:39:49.937319 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:49.937467 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.61:44305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260924 11:39:49.937919 29859 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.212.61:44305: connect: Connection refused (error 111)
I20260924 11:39:49.939695 29912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:49.939223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:44305 (local address 127.0.0.1:51966)
0924 11:39:49.939436 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:49.939526 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.61:44305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 11:39:49.939894 29909 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.212.61:44305: Network error: Client connection negotiation failed: client connection to 127.6.212.61:44305: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.61.keytab.
Entry for principal kudu/127.6.212.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.61.keytab.
I20260924 11:39:49.968520 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.61:44305
--webserver_interface=127.6.212.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.61.keytab
--principal=kudu/127.6.212.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:33451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:36299,127.6.212.61:44305,127.6.212.60:34435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:39:49.989228 29909 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.212.61" port: 44305 } attempt: 1
I20260924 11:39:49.990473 29912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:49.990027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:44305 (local address 127.0.0.1:51976)
0924 11:39:49.990230 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:49.990332 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.61:44305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 11:39:49.991498 29912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:49.991077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:44305 (local address 127.0.0.1:51988)
0924 11:39:49.991225 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:49.991286 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.61:44305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 11:39:49.991708 29909 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.212.61:44305: Network error: Client connection negotiation failed: client connection to 127.6.212.61:44305: connect: Connection refused (error 111)
I20260924 11:39:50.059954 29909 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.212.61" port: 44305 } attempt: 2
I20260924 11:39:50.061219 29912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:50.060771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:44305 (local address 127.0.0.1:51992)
0924 11:39:50.060975 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:50.061076 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.61:44305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260924 11:39:50.062168 29912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:50.061822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:44305 (local address 127.0.0.1:52000)
0924 11:39:50.061960 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:50.062045 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.61:44305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 11:39:50.062358 29909 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.212.61:44305: Network error: Client connection negotiation failed: client connection to 127.6.212.61:44305: connect: Connection refused (error 111)
I20260924 11:39:50.153630 29909 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.212.61" port: 44305 } attempt: 3
I20260924 11:39:50.154793 29912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:50.154305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:44305 (local address 127.0.0.1:52016)
0924 11:39:50.154569 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:50.154655 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.61:44305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260924 11:39:50.155782 29912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:50.155451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:44305 (local address 127.0.0.1:52026)
0924 11:39:50.155586 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:50.155657 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.61:44305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 11:39:50.155967 29909 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.212.61:44305: Network error: Client connection negotiation failed: client connection to 127.6.212.61:44305: connect: Connection refused (error 111)
I20260924 11:39:50.299216 29909 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.212.61" port: 44305 } attempt: 4
I20260924 11:39:50.300509 29912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:50.299934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:44305 (local address 127.0.0.1:52030)
0924 11:39:50.300221 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:50.300312 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.61:44305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
W20260924 11:39:50.301189 29916 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
I20260924 11:39:50.301560 29912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:50.301152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:44305 (local address 127.0.0.1:52046)
0924 11:39:50.301294 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:50.301390 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.61:44305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 11:39:50.301578 29916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:39:50.301687 29916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:39:50.301748 29909 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.212.61:44305: Network error: Client connection negotiation failed: client connection to 127.6.212.61:44305: connect: Connection refused (error 111)
W20260924 11:39:50.301851 29916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 11:39:50.309458 29916 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:39:50.309593 29916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:39:50.309724 29916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:39:50.309813 29916 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:39:50.309935 29916 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:39:50.320181 29916 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:33451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:36299,127.6.212.61:44305,127.6.212.60:34435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.61.keytab
--principal=kudu/127.6.212.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.61:44305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.212.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:39:50.321586 29916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:39:50.322949 29916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 11:39:50.334940 29921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:50.334982 29922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:50.334867 29924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:39:50.336357 29916 server_base.cc:1034] running on GCE node
I20260924 11:39:50.337121 29916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:39:50.338567 29916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:39:50.339812 29916 hybrid_clock.cc:648] HybridClock initialized: now 1790249990339753 us; error 33 us; skew 500 ppm
I20260924 11:39:50.340152 29916 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:39:50 dist-test-slave-cq7c krb5kdc[29829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249990, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:39:50.343323 29916 init.cc:377] Logged in from keytab as kudu/127.6.212.61@KRBTEST.COM (short username kudu)
I20260924 11:39:50.344918 29916 webserver.cc:460] Webserver started at http://127.6.212.61:40407/ using document root <none> and password file <none>
I20260924 11:39:50.345583 29916 fs_manager.cc:359] Metadata directory not provided
I20260924 11:39:50.345692 29916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:39:50.346021 29916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:39:50.349028 29916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-1/data/instance:
uuid: "b4561d9020204a5e9832843c71a4e7ca"
format_stamp: "Formatted at 2026-09-24 11:39:50 on dist-test-slave-cq7c"
server_key: "f8b3751d8d8c6d8aaec58fa6da37965c"
server_key_iv: "f793ecbdb0f0d6d153578117b4b1e252"
server_key_version: "encryptionkey@0"
I20260924 11:39:50.349764 29916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-1/wal/instance:
uuid: "b4561d9020204a5e9832843c71a4e7ca"
format_stamp: "Formatted at 2026-09-24 11:39:50 on dist-test-slave-cq7c"
server_key: "f8b3751d8d8c6d8aaec58fa6da37965c"
server_key_iv: "f793ecbdb0f0d6d153578117b4b1e252"
server_key_version: "encryptionkey@0"
I20260924 11:39:50.354715 29916 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 11:39:50.358258 29931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:39:50.359697 29916 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 11:39:50.359869 29916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-1/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-1/wal
uuid: "b4561d9020204a5e9832843c71a4e7ca"
format_stamp: "Formatted at 2026-09-24 11:39:50 on dist-test-slave-cq7c"
server_key: "f8b3751d8d8c6d8aaec58fa6da37965c"
server_key_iv: "f793ecbdb0f0d6d153578117b4b1e252"
server_key_version: "encryptionkey@0"
I20260924 11:39:50.360092 29916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:39:50.373718 29916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:39:50.378027 29916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:39:50.378309 29916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:39:50.397480 29916 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.61:44305
I20260924 11:39:50.397500 29982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.61:44305 every 8 connection(s)
I20260924 11:39:50.398952 29916 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-1/data/info.pb
I20260924 11:39:50.403188 29983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:39:50.406001 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 29916
I20260924 11:39:50.406188 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-1/wal/instance
I20260924 11:39:50.406733 6992 external_mini_cluster.cc:1428] Setting key d2995f37a7a647a084efa58cf01dbc76
I20260924 11:39:50.412062 29983 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36299 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:39:50 dist-test-slave-cq7c krb5kdc[29829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249990, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.61@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:39:50.437855 29911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:50.415425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35260 (local address 127.6.212.62:36299)
0924 11:39:50.415637 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:50.415644 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:50.416964 (+ 1320us) server_negotiation.cc:407] Connection header received
0924 11:39:50.419623 (+ 2659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:50.419633 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:50.419760 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:50.419929 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:50.421440 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:50.422788 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:50.424502 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:50.425420 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:50.425593 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:50.429486 (+ 3893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:50.429497 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:50.429506 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:39:50.429561 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:50.434054 (+ 4493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:50.434858 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:50.434868 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:50.434876 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:50.435067 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:50.435583 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:50.435593 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:50.435600 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:50.436112 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:50.436316 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:50.437054 (+ 738us) server_negotiation.cc:299] Negotiation successful
0924 11:39:50.437365 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 11:39:50.438375 29985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:50.415929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36299 (local address 127.0.0.1:35260)
0924 11:39:50.416768 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:50.416823 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 11:39:50.419195 (+ 2372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:39:50.420134 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:39:50.420143 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:39:50.420678 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:39:50.421242 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:50.421259 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:50.422968 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:50.422972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:50.424332 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:50.424344 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:50.425647 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:50.425652 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:50.425859 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:50.426632 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:39:50.426667 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:39:50.429219 (+ 2552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:39:50.434217 (+ 4998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:39:50.434224 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:39:50.434237 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:39:50.434691 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:39:50.435194 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:39:50.435197 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:39:50.435199 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:39:50.435447 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:39:50.436253 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:39:50.436259 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:39:50.436631 (+ 372us) client_negotiation.cc:769] Sending connection context
0924 11:39:50.436855 (+ 224us) client_negotiation.cc:240] Negotiation successful
0924 11:39:50.437232 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":315,"threads_started":1}
WARNING: no policy specified for kudu/127.6.212.60@KRBTEST.COM; defaulting to no policy
I20260924 11:39:50.440446 29983 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.212.61" port: 44305 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.212.60@KRBTEST.COM" created.
Sep 24 11:39:50 dist-test-slave-cq7c krb5kdc[29829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249990, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.61@KRBTEST.COM for kudu/127.6.212.61@KRBTEST.COM
I20260924 11:39:50.457366 29985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:50.441230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:44305 (local address 127.0.0.1:52050)
0924 11:39:50.441472 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:50.441487 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 11:39:50.441573 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:39:50.442512 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:39:50.442519 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:39:50.443062 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:39:50.443269 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:50.443282 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:50.444745 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:50.444754 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:50.446221 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:50.446238 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:50.447071 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:50.447083 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:50.447291 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:50.447899 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:39:50.447933 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:39:50.450497 (+ 2564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:39:50.454320 (+ 3823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:39:50.454326 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:39:50.454332 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:39:50.454777 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:39:50.455223 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:39:50.455229 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:39:50.455233 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:39:50.455469 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:39:50.456175 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:39:50.456183 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:39:50.456362 (+ 179us) client_negotiation.cc:769] Sending connection context
0924 11:39:50.456520 (+ 158us) client_negotiation.cc:240] Negotiation successful
0924 11:39:50.456864 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260924 11:39:50.457501 29987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:50.441408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52050 (local address 127.6.212.61:44305)
0924 11:39:50.442029 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:50.442035 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:50.442056 (+ 21us) server_negotiation.cc:407] Connection header received
0924 11:39:50.442094 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:50.442099 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:50.442335 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:50.442505 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:50.443551 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:50.444607 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:50.446363 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:50.446907 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:50.447000 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:50.450771 (+ 3771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:50.450784 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:50.450799 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:39:50.450837 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:50.454142 (+ 3305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:50.454939 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:50.454957 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:50.454959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:50.455087 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:50.455611 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:50.455619 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:50.455629 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:50.456012 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:50.456254 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:50.456711 (+ 457us) server_negotiation.cc:299] Negotiation successful
0924 11:39:50.456995 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":482,"thread_start_us":356,"threads_started":1}
I20260924 11:39:50.460325 29983 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.212.60" port: 34435 } has no permanent_uuid. Determining permanent_uuid...
I20260924 11:39:50.461704 29985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:50.461113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:34435 (local address 127.0.0.1:39494)
0924 11:39:50.461339 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:50.461506 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:34435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260924 11:39:50.462009 29934 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.212.60:34435: connect: Connection refused (error 111)
I20260924 11:39:50.463971 29985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:50.463512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:34435 (local address 127.0.0.1:39510)
0924 11:39:50.463688 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:50.463804 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:34435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 11:39:50.464345 29983 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.212.60:34435: Network error: Client connection negotiation failed: client connection to 127.6.212.60:34435: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.212.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.60.keytab.
Entry for principal kudu/127.6.212.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.60.keytab.
I20260924 11:39:50.479091 6992 external_mini_cluster.cc:1326] Running /tmp/dist-test-task75Snja/build/asan/bin/kudu
/tmp/dist-test-task75Snja/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.60:34435
--webserver_interface=127.6.212.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.60.keytab
--principal=kudu/127.6.212.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.212.20:33451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.212.62:36299,127.6.212.61:44305,127.6.212.60:34435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 11:39:50.513756 29983 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.212.60" port: 34435 } attempt: 1
I20260924 11:39:50.514962 29985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:50.514511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:34435 (local address 127.0.0.1:39526)
0924 11:39:50.514708 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:50.514816 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:34435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260924 11:39:50.515975 29985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:50.515648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:34435 (local address 127.0.0.1:39530)
0924 11:39:50.515804 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:50.515878 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:34435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 11:39:50.516192 29983 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.212.60:34435: Network error: Client connection negotiation failed: client connection to 127.6.212.60:34435: connect: Connection refused (error 111)
I20260924 11:39:50.584425 29983 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.212.60" port: 34435 } attempt: 2
I20260924 11:39:50.585554 29985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:50.585106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:34435 (local address 127.0.0.1:39536)
0924 11:39:50.585300 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:50.585397 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:34435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 11:39:50.586539 29985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:50.586206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:34435 (local address 127.0.0.1:39552)
0924 11:39:50.586348 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:50.586421 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:34435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 11:39:50.586759 29983 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.212.60:34435: Network error: Client connection negotiation failed: client connection to 127.6.212.60:34435: connect: Connection refused (error 111)
I20260924 11:39:50.601001 29909 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.212.61" port: 44305 } attempt: 5
Sep 24 11:39:50 dist-test-slave-cq7c krb5kdc[29829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249989, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for kudu/127.6.212.61@KRBTEST.COM
I20260924 11:39:50.615383 29912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:50.601693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:44305 (local address 127.0.0.1:52052)
0924 11:39:50.601874 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:50.601906 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 11:39:50.602008 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:39:50.602447 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:39:50.602452 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:39:50.602918 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:39:50.603135 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:50.603145 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:50.604584 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:50.604588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:50.605498 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:50.605511 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:50.606583 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:50.606588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:50.606751 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:50.607444 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:39:50.607477 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:39:50.609511 (+ 2034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:39:50.612843 (+ 3332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:39:50.612849 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:39:50.612853 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:39:50.613278 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:39:50.613660 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:39:50.613666 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:39:50.613669 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:39:50.613804 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:39:50.614374 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:39:50.614379 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:39:50.614535 (+ 156us) client_negotiation.cc:769] Sending connection context
0924 11:39:50.614705 (+ 170us) client_negotiation.cc:240] Negotiation successful
0924 11:39:50.615013 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 11:39:50.615355 29987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:50.601788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52052 (local address 127.6.212.61:44305)
0924 11:39:50.601968 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:50.601974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:50.601998 (+ 24us) server_negotiation.cc:407] Connection header received
0924 11:39:50.602147 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:50.602153 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:50.602234 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:50.602355 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:50.603311 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:50.604419 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:50.605654 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:50.606377 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:50.606552 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:50.609716 (+ 3164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:50.609723 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:50.609728 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:39:50.609767 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:50.612697 (+ 2930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:50.613399 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:50.613406 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:50.613410 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:50.613544 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:50.613961 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:50.613966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:50.613971 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:50.614254 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:50.614409 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:50.614809 (+ 400us) server_negotiation.cc:299] Negotiation successful
0924 11:39:50.615028 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 11:39:50.616500 29909 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.212.60" port: 34435 } has no permanent_uuid. Determining permanent_uuid...
I20260924 11:39:50.617679 29912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:50.617099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:34435 (local address 127.0.0.1:39568)
0924 11:39:50.617328 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:50.617457 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:34435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260924 11:39:50.618686 29912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:50.618259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:34435 (local address 127.0.0.1:39572)
0924 11:39:50.618461 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:50.618563 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:34435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260924 11:39:50.618906 29909 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.212.60:34435: Network error: Client connection negotiation failed: client connection to 127.6.212.60:34435: connect: Connection refused (error 111)
I20260924 11:39:50.670158 29909 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.212.60" port: 34435 } attempt: 1
I20260924 11:39:50.671458 29912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:50.670863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:34435 (local address 127.0.0.1:39588)
0924 11:39:50.671082 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:50.671194 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:34435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 11:39:50.672514 29912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:50.672117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:34435 (local address 127.0.0.1:39592)
0924 11:39:50.672276 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:50.672361 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:34435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 11:39:50.672772 29909 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.212.60:34435: Network error: Client connection negotiation failed: client connection to 127.6.212.60:34435: connect: Connection refused (error 111)
I20260924 11:39:50.677995 29983 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.212.60" port: 34435 } attempt: 3
I20260924 11:39:50.679201 29985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:50.678703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:34435 (local address 127.0.0.1:39594)
0924 11:39:50.678926 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:50.679038 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:34435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260924 11:39:50.680298 29985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:50.679868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:34435 (local address 127.0.0.1:39602)
0924 11:39:50.680114 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:50.680187 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:34435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
W20260924 11:39:50.680578 29983 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.212.60:34435: Network error: Client connection negotiation failed: client connection to 127.6.212.60:34435: connect: Connection refused (error 111)
I20260924 11:39:50.741014 29909 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.212.60" port: 34435 } attempt: 2
I20260924 11:39:50.742324 29912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:50.741725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:34435 (local address 127.0.0.1:39612)
0924 11:39:50.741989 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:50.742112 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:34435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260924 11:39:50.743446 29912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:50.743020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:34435 (local address 127.0.0.1:39624)
0924 11:39:50.743192 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:50.743286 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:34435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 11:39:50.743676 29909 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.212.60:34435: Network error: Client connection negotiation failed: client connection to 127.6.212.60:34435: connect: Connection refused (error 111)
W20260924 11:39:50.819051 29990 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 11:39:50.819473 29990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 11:39:50.819630 29990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 11:39:50.819763 29990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 11:39:50.823817 29983 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.212.60" port: 34435 } attempt: 4
I20260924 11:39:50.825197 29985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:50.824623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:34435 (local address 127.0.0.1:39626)
0924 11:39:50.824863 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:50.824978 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:34435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260924 11:39:50.826303 29985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:50.825888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:34435 (local address 127.0.0.1:39638)
0924 11:39:50.826081 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:50.826168 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:34435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260924 11:39:50.826509 29983 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.212.60:34435: Network error: Client connection negotiation failed: client connection to 127.6.212.60:34435: connect: Connection refused (error 111)
W20260924 11:39:50.828006 29990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 11:39:50.828080 29990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 11:39:50.828194 29990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 11:39:50.828269 29990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 11:39:50.828374 29990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 11:39:50.838590 29990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.212.20:33451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.212.62:36299,127.6.212.61:44305,127.6.212.60:34435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/krb5kdc/kudu_127.6.212.60.keytab
--principal=kudu/127.6.212.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.212.60:34435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.212.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 10:48:45 UTC on ccycloud.quasar-ropeul.root.comops.site
build id 4669759
ASAN enabled
I20260924 11:39:50.840037 29990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 11:39:50.841375 29990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 11:39:50.849885 29909 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.212.60" port: 34435 } attempt: 3
I20260924 11:39:50.851049 29912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:50.850568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:34435 (local address 127.0.0.1:39644)
0924 11:39:50.850746 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:50.850843 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:34435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 11:39:50.852711 29912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:50.852016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:34435 (local address 127.0.0.1:39646)
0924 11:39:50.852396 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:50.852516 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.212.60:34435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248}
W20260924 11:39:50.853153 29909 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.212.60:34435: Network error: Client connection negotiation failed: client connection to 127.6.212.60:34435: connect: Connection refused (error 111)
W20260924 11:39:50.853173 29996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:50.853127 29995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 11:39:50.853302 29998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 11:39:50.855343 29990 server_base.cc:1034] running on GCE node
I20260924 11:39:50.856248 29990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 11:39:50.857789 29990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 11:39:50.859064 29990 hybrid_clock.cc:648] HybridClock initialized: now 1790249990858968 us; error 57 us; skew 500 ppm
I20260924 11:39:50.859516 29990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 11:39:50 dist-test-slave-cq7c krb5kdc[29829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249990, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 11:39:50.864032 29990 init.cc:377] Logged in from keytab as kudu/127.6.212.60@KRBTEST.COM (short username kudu)
I20260924 11:39:50.866007 29990 webserver.cc:460] Webserver started at http://127.6.212.60:43925/ using document root <none> and password file <none>
I20260924 11:39:50.866776 29990 fs_manager.cc:359] Metadata directory not provided
I20260924 11:39:50.866894 29990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 11:39:50.867225 29990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 11:39:50.870915 29990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-2/data/instance:
uuid: "43f24860daab40238daa7439e3d68c1f"
format_stamp: "Formatted at 2026-09-24 11:39:50 on dist-test-slave-cq7c"
server_key: "d1bff41faa51b89aa581ca602a7fd028"
server_key_iv: "24f82e74b1dcd12cc6bcd7b818df3c0c"
server_key_version: "encryptionkey@0"
I20260924 11:39:50.871930 29990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-2/wal/instance:
uuid: "43f24860daab40238daa7439e3d68c1f"
format_stamp: "Formatted at 2026-09-24 11:39:50 on dist-test-slave-cq7c"
server_key: "d1bff41faa51b89aa581ca602a7fd028"
server_key_iv: "24f82e74b1dcd12cc6bcd7b818df3c0c"
server_key_version: "encryptionkey@0"
I20260924 11:39:50.878077 29990 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260924 11:39:50.881943 30005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 11:39:50.883443 29990 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 11:39:50.883605 29990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-2/data,/tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-2/wal
uuid: "43f24860daab40238daa7439e3d68c1f"
format_stamp: "Formatted at 2026-09-24 11:39:50 on dist-test-slave-cq7c"
server_key: "d1bff41faa51b89aa581ca602a7fd028"
server_key_iv: "24f82e74b1dcd12cc6bcd7b818df3c0c"
server_key_version: "encryptionkey@0"
I20260924 11:39:50.883807 29990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 11:39:50.896549 29990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 11:39:50.901876 29990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 11:39:50.902144 29990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 11:39:50.922410 29990 rpc_server.cc:307] RPC server started. Bound to: 127.6.212.60:34435
I20260924 11:39:50.922462 30056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.212.60:34435 every 8 connection(s)
I20260924 11:39:50.923959 29990 server_base.cc:1166] Dumped server information to /tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-2/data/info.pb
I20260924 11:39:50.927292 6992 external_mini_cluster.cc:1388] Started /tmp/dist-test-task75Snja/build/asan/bin/kudu as pid 29990
I20260924 11:39:50.927616 6992 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task75Snja/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790249617057452-6992-0/minicluster-data/master-2/wal/instance
I20260924 11:39:50.927953 30057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 11:39:50.928220 6992 external_mini_cluster.cc:1428] Setting key fb95de35807b92b08fabe04a0055fa02
I20260924 11:39:50.936904 30057 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36299 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:39:50 dist-test-slave-cq7c krb5kdc[29829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.60@KRBTEST.COM
Sep 24 11:39:50 dist-test-slave-cq7c krb5kdc[29829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249990, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.60@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
I20260924 11:39:50.953373 30060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:50.930270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39660 (local address 127.6.212.60:34435)
0924 11:39:50.931015 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:50.931029 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:50.931069 (+ 40us) server_negotiation.cc:407] Connection header received
0924 11:39:50.932996 (+ 1927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:50.933019 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:50.933427 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:50.933844 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:50.935304 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:50.936625 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:50.937944 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:50.938822 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:50.939015 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:50.943843 (+ 4828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:50.943853 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:50.943868 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:39:50.943915 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:50.947892 (+ 3977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:50.948843 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:50.948852 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:50.948861 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:50.949094 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:50.949631 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:50.949636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:50.949639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:50.950380 (+ 741us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:50.950669 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:50.951244 (+ 575us) server_negotiation.cc:299] Negotiation successful
0924 11:39:50.951825 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":329,"threads_started":1}
I20260924 11:39:50.956570 30061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:50.939336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.62:36299 (local address 127.0.0.1:35268)
0924 11:39:50.940068 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:50.940111 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 11:39:50.940275 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:39:50.940714 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:39:50.940727 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:39:50.941379 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:39:50.941642 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:50.941651 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:50.943280 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:50.943284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:50.944870 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:50.944887 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:50.945669 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:50.945676 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:50.945889 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:50.946488 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:39:50.946519 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:39:50.949492 (+ 2973us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:39:50.953375 (+ 3883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:39:50.953384 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:39:50.953390 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:39:50.954077 (+ 687us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:39:50.954538 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:39:50.954543 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:39:50.954546 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:39:50.954728 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:39:50.955559 (+ 831us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:39:50.955565 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:39:50.955710 (+ 145us) client_negotiation.cc:769] Sending connection context
0924 11:39:50.955852 (+ 142us) client_negotiation.cc:240] Negotiation successful
0924 11:39:50.956137 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":563,"threads_started":1}
I20260924 11:39:50.957036 30062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:50.939417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35268 (local address 127.6.212.62:36299)
0924 11:39:50.940017 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:50.940026 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:50.940194 (+ 168us) server_negotiation.cc:407] Connection header received
0924 11:39:50.940417 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:50.940422 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:50.940497 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:50.940648 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:50.941790 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:50.943108 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:50.945027 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:50.945433 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:50.945615 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:50.949702 (+ 4087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:50.949712 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:50.949718 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:39:50.949760 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:50.953217 (+ 3457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:50.954241 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:50.954245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:50.954248 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:50.954377 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:50.954864 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:50.954869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:50.954871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:50.955310 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:50.955524 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:50.956022 (+ 498us) server_negotiation.cc:299] Negotiation successful
0924 11:39:50.956268 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":280,"threads_started":1}
I20260924 11:39:50.958297 30057 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.212.61" port: 44305 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:39:50 dist-test-slave-cq7c krb5kdc[29829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.62@KRBTEST.COM
Sep 24 11:39:50 dist-test-slave-cq7c krb5kdc[29829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249990, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.60@KRBTEST.COM for kudu/127.6.212.61@KRBTEST.COM
I20260924 11:39:50.971477 30062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:50.957125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35282 (local address 127.6.212.62:36299)
0924 11:39:50.957318 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:50.957323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:50.957341 (+ 18us) server_negotiation.cc:407] Connection header received
0924 11:39:50.957429 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:50.957434 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:50.957541 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:50.958723 (+ 1182us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:50.958878 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:50.959761 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:50.961066 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:50.961481 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:50.961614 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:50.966071 (+ 4457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:50.966078 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:50.966082 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:39:50.966119 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:50.968815 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:50.969562 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:50.969566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:50.969568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:50.969675 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:50.970073 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:50.970076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:50.970078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:50.970329 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:50.970604 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:50.970877 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 11:39:50.971097 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":443,"threads_started":1}
I20260924 11:39:50.971475 30061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:50.959022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.61:44305 (local address 127.0.0.1:52064)
0924 11:39:50.959210 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:50.959228 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:39:50.959320 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:39:50.959897 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:39:50.959901 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:39:50.960257 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:39:50.960386 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:50.960414 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:50.961367 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:50.961375 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:50.962213 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:50.962223 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:50.962956 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:50.962963 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:50.963124 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:50.963731 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:39:50.963747 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:39:50.966069 (+ 2322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:39:50.969011 (+ 2942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:39:50.969014 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:39:50.969017 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:39:50.969404 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:39:50.969778 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:39:50.969785 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:39:50.969791 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:39:50.969934 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:39:50.970432 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:39:50.970436 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:39:50.970682 (+ 246us) client_negotiation.cc:769] Sending connection context
0924 11:39:50.970848 (+ 166us) client_negotiation.cc:240] Negotiation successful
0924 11:39:50.971096 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 11:39:50.972015 29987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:50.959044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52064 (local address 127.6.212.61:44305)
0924 11:39:50.959250 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:50.959258 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:50.959277 (+ 19us) server_negotiation.cc:407] Connection header received
0924 11:39:50.959699 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:50.959703 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:50.959767 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:50.959904 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:50.960575 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:50.961237 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:50.962349 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:50.962762 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:50.962889 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:50.966233 (+ 3344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:50.966238 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:50.966241 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:39:50.966269 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:50.968834 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:50.969525 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:50.969530 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:50.969534 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:50.969669 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:50.970042 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:50.970048 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:50.970054 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:50.970310 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:50.970448 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:50.970966 (+ 518us) server_negotiation.cc:299] Negotiation successful
0924 11:39:50.971178 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260924 11:39:50.972983 30057 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.212.60" port: 34435 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 11:39:50 dist-test-slave-cq7c krb5kdc[29829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249990, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.60@KRBTEST.COM for kudu/127.6.212.60@KRBTEST.COM
Sep 24 11:39:50 dist-test-slave-cq7c krb5kdc[29829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.212.61@KRBTEST.COM
I20260924 11:39:50.985735 30060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:50.973806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39666 (local address 127.6.212.60:34435)
0924 11:39:50.973998 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:50.974002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:50.974014 (+ 12us) server_negotiation.cc:407] Connection header received
0924 11:39:50.974045 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:50.974048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:50.974093 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:50.974190 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:50.974826 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:50.975558 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:50.976649 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:50.977035 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:50.977173 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:50.980220 (+ 3047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:50.980225 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:50.980228 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:39:50.980254 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:50.983113 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:50.983793 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:50.983800 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:50.983809 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:50.983915 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:50.984259 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:50.984263 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:50.984267 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:50.984529 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:50.984675 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:50.985022 (+ 347us) server_negotiation.cc:299] Negotiation successful
0924 11:39:50.985397 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":168,"server-negotiator.queue_time_us":49}
I20260924 11:39:50.985742 30061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:50.973612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:34435 (local address 127.0.0.1:39666)
0924 11:39:50.973759 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:50.973777 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 11:39:50.973867 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:39:50.974202 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:39:50.974206 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:39:50.974584 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:39:50.974699 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:50.974706 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:50.975669 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:50.975676 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:50.976516 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:50.976526 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:50.977186 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:50.977191 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:50.977393 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:50.978000 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:39:50.978020 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:39:50.980050 (+ 2030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:39:50.983248 (+ 3198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:39:50.983256 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:39:50.983260 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:39:50.983675 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:39:50.984022 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:39:50.984028 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:39:50.984031 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:39:50.984151 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:39:50.984639 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:39:50.984644 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:39:50.984778 (+ 134us) client_negotiation.cc:769] Sending connection context
0924 11:39:50.984933 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 11:39:50.985248 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":8}
I20260924 11:39:50.988839 29987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:50.973529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52080 (local address 127.6.212.61:44305)
0924 11:39:50.973692 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:50.973695 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:50.973707 (+ 12us) server_negotiation.cc:407] Connection header received
0924 11:39:50.973742 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:50.973745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:50.973791 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:50.973892 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:50.975158 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:50.976072 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:50.977472 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:50.978134 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:50.978295 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:50.983246 (+ 4951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:50.983251 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:50.983254 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:39:50.983284 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:50.985961 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:50.986696 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:50.986702 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:50.986705 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:50.986862 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:50.987253 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:50.987260 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:50.987263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:50.987673 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:50.987966 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:50.988256 (+ 290us) server_negotiation.cc:299] Negotiation successful
0924 11:39:50.988489 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 11:39:50.991498 30057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43f24860daab40238daa7439e3d68c1f: Bootstrap starting.
I20260924 11:39:50.993963 6992 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 11:39:50.995491 30057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43f24860daab40238daa7439e3d68c1f: Neither blocks nor log segments found. Creating new log.
I20260924 11:39:50.996837 30057 log.cc:824] T 00000000000000000000000000000000 P 43f24860daab40238daa7439e3d68c1f: Log is configured to *not* fsync() on all Append() calls
I20260924 11:39:50.999828 30057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43f24860daab40238daa7439e3d68c1f: No bootstrap required, opened a new log
I20260924 11:39:50.997080 29884 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "95e2baa7c71c40c785fcf6d604c22eee"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35282
I20260924 11:39:51.006915 30057 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43f24860daab40238daa7439e3d68c1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e2baa7c71c40c785fcf6d604c22eee" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36299 } } peers { permanent_uuid: "b4561d9020204a5e9832843c71a4e7ca" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 44305 } } peers { permanent_uuid: "43f24860daab40238daa7439e3d68c1f" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 34435 } }
I20260924 11:39:51.007252 30057 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43f24860daab40238daa7439e3d68c1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43f24860daab40238daa7439e3d68c1f, State: Initialized, Role: FOLLOWER
I20260924 11:39:51.007808 30057 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43f24860daab40238daa7439e3d68c1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e2baa7c71c40c785fcf6d604c22eee" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36299 } } peers { permanent_uuid: "b4561d9020204a5e9832843c71a4e7ca" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 44305 } } peers { permanent_uuid: "43f24860daab40238daa7439e3d68c1f" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 34435 } }
I20260924 11:39:51.008718 30065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43f24860daab40238daa7439e3d68c1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e2baa7c71c40c785fcf6d604c22eee" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36299 } } peers { permanent_uuid: "b4561d9020204a5e9832843c71a4e7ca" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 44305 } } peers { permanent_uuid: "43f24860daab40238daa7439e3d68c1f" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 34435 } } }
I20260924 11:39:51.009071 30065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43f24860daab40238daa7439e3d68c1f [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:39:51.010113 30057 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43f24860daab40238daa7439e3d68c1f [sys.catalog]: configured and running, proceeding with master startup.
W20260924 11:39:51.019942 30076 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 43f24860daab40238daa7439e3d68c1f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 11:39:51.020139 30076 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 11:39:51.017728 29990 master_runner.cc:186] Error getting master registration for 127.6.212.62:36299: OK, instance_id { permanent_uuid: "95e2baa7c71c40c785fcf6d604c22eee" instance_seqno: 1790249989893413 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 11:39:51.028033 29990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 11:39:51.030380 29909 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.212.60" port: 34435 } attempt: 4
Sep 24 11:39:51 dist-test-slave-cq7c krb5kdc[29829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249989, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.62@KRBTEST.COM for kudu/127.6.212.60@KRBTEST.COM
I20260924 11:39:51.046298 29912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:51.031133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:34435 (local address 127.0.0.1:39674)
0924 11:39:51.031317 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:51.031337 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 11:39:51.031503 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:39:51.031835 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:39:51.031839 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:39:51.032265 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:39:51.032422 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:51.032431 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:51.033477 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:51.033483 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:51.034422 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:51.034434 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:51.035224 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:51.035230 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:51.035482 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:51.036142 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:39:51.036165 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:39:51.039024 (+ 2859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:39:51.043403 (+ 4379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:39:51.043408 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:39:51.043413 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:39:51.043953 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:39:51.044401 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:39:51.044406 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:39:51.044410 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:39:51.044582 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:39:51.045258 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:39:51.045263 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:39:51.045432 (+ 169us) client_negotiation.cc:769] Sending connection context
0924 11:39:51.045574 (+ 142us) client_negotiation.cc:240] Negotiation successful
0924 11:39:51.045889 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 11:39:51.046732 30060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:51.031290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39674 (local address 127.6.212.60:34435)
0924 11:39:51.031512 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:51.031518 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:51.031540 (+ 22us) server_negotiation.cc:407] Connection header received
0924 11:39:51.031611 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:51.031616 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:51.031695 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:51.031799 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:51.032573 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:51.033317 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:51.034572 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:51.035070 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:51.035208 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:51.039290 (+ 4082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:51.039298 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:51.039305 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:39:51.039347 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:51.043208 (+ 3861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:51.044095 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:51.044105 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:51.044109 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:51.044279 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:51.044706 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:51.044711 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:51.044714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:51.045121 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:51.045294 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:51.045786 (+ 492us) server_negotiation.cc:299] Negotiation successful
0924 11:39:51.046023 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 11:39:51.052103 29909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95e2baa7c71c40c785fcf6d604c22eee: Bootstrap starting.
I20260924 11:39:51.055202 29909 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95e2baa7c71c40c785fcf6d604c22eee: Neither blocks nor log segments found. Creating new log.
I20260924 11:39:51.056263 29909 log.cc:824] T 00000000000000000000000000000000 P 95e2baa7c71c40c785fcf6d604c22eee: Log is configured to *not* fsync() on all Append() calls
I20260924 11:39:51.059998 29909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95e2baa7c71c40c785fcf6d604c22eee: No bootstrap required, opened a new log
I20260924 11:39:51.067909 29909 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95e2baa7c71c40c785fcf6d604c22eee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e2baa7c71c40c785fcf6d604c22eee" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36299 } } peers { permanent_uuid: "b4561d9020204a5e9832843c71a4e7ca" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 44305 } } peers { permanent_uuid: "43f24860daab40238daa7439e3d68c1f" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 34435 } }
I20260924 11:39:51.068348 29909 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95e2baa7c71c40c785fcf6d604c22eee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95e2baa7c71c40c785fcf6d604c22eee, State: Initialized, Role: FOLLOWER
I20260924 11:39:51.068878 29909 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95e2baa7c71c40c785fcf6d604c22eee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e2baa7c71c40c785fcf6d604c22eee" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36299 } } peers { permanent_uuid: "b4561d9020204a5e9832843c71a4e7ca" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 44305 } } peers { permanent_uuid: "43f24860daab40238daa7439e3d68c1f" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 34435 } }
I20260924 11:39:51.070135 30079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95e2baa7c71c40c785fcf6d604c22eee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e2baa7c71c40c785fcf6d604c22eee" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36299 } } peers { permanent_uuid: "b4561d9020204a5e9832843c71a4e7ca" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 44305 } } peers { permanent_uuid: "43f24860daab40238daa7439e3d68c1f" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 34435 } } }
I20260924 11:39:51.070752 30079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95e2baa7c71c40c785fcf6d604c22eee [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:39:51.071514 29909 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95e2baa7c71c40c785fcf6d604c22eee [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:39:51.071619 29884 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95e2baa7c71c40c785fcf6d604c22eee [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 11:39:51.071805 29884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95e2baa7c71c40c785fcf6d604c22eee [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:39:51.076133 29884 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95e2baa7c71c40c785fcf6d604c22eee [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e2baa7c71c40c785fcf6d604c22eee" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36299 } } peers { permanent_uuid: "b4561d9020204a5e9832843c71a4e7ca" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 44305 } } peers { permanent_uuid: "43f24860daab40238daa7439e3d68c1f" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 34435 } }
I20260924 11:39:51.080390 29884 leader_election.cc:288] T 00000000000000000000000000000000 P 95e2baa7c71c40c785fcf6d604c22eee [CANDIDATE]: Term 1 election: Requested vote from peers b4561d9020204a5e9832843c71a4e7ca (127.6.212.61:44305), 43f24860daab40238daa7439e3d68c1f (127.6.212.60:34435)
I20260924 11:39:51.080703 30032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "95e2baa7c71c40c785fcf6d604c22eee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "43f24860daab40238daa7439e3d68c1f"
I20260924 11:39:51.081254 30032 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43f24860daab40238daa7439e3d68c1f [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:39:51.086712 30032 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 43f24860daab40238daa7439e3d68c1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95e2baa7c71c40c785fcf6d604c22eee in term 1.
I20260924 11:39:51.088433 29860 leader_election.cc:302] T 00000000000000000000000000000000 P 95e2baa7c71c40c785fcf6d604c22eee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43f24860daab40238daa7439e3d68c1f, 95e2baa7c71c40c785fcf6d604c22eee; no voters:
I20260924 11:39:51.089018 30079 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95e2baa7c71c40c785fcf6d604c22eee [term 1 FOLLOWER]: Leader election won for term 1
I20260924 11:39:51.103348 30079 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95e2baa7c71c40c785fcf6d604c22eee [term 1 LEADER]: Becoming Leader. State: Replica: 95e2baa7c71c40c785fcf6d604c22eee, State: Running, Role: LEADER
I20260924 11:39:51.104107 30079 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95e2baa7c71c40c785fcf6d604c22eee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e2baa7c71c40c785fcf6d604c22eee" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36299 } } peers { permanent_uuid: "b4561d9020204a5e9832843c71a4e7ca" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 44305 } } peers { permanent_uuid: "43f24860daab40238daa7439e3d68c1f" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 34435 } }
I20260924 11:39:51.119778 30092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95e2baa7c71c40c785fcf6d604c22eee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95e2baa7c71c40c785fcf6d604c22eee. Latest consensus state: current_term: 1 leader_uuid: "95e2baa7c71c40c785fcf6d604c22eee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e2baa7c71c40c785fcf6d604c22eee" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36299 } } peers { permanent_uuid: "b4561d9020204a5e9832843c71a4e7ca" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 44305 } } peers { permanent_uuid: "43f24860daab40238daa7439e3d68c1f" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 34435 } } }
I20260924 11:39:51.120244 30092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95e2baa7c71c40c785fcf6d604c22eee [sys.catalog]: This master's current role is: LEADER
I20260924 11:39:51.125953 29983 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.212.60" port: 34435 } attempt: 5
W20260924 11:39:51.133487 29863 tablet.cc:2369] T 00000000000000000000000000000000 P 95e2baa7c71c40c785fcf6d604c22eee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:39:51.144582 29987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:51.105103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52090 (local address 127.6.212.61:44305)
0924 11:39:51.105299 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:51.105306 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:51.110783 (+ 5477us) server_negotiation.cc:407] Connection header received
0924 11:39:51.111969 (+ 1186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:51.111975 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:51.112081 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:51.112156 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:51.116628 (+ 4472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:51.117739 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:51.122929 (+ 5190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:51.123705 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:51.123812 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:51.130860 (+ 7048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:51.130868 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:51.130872 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:39:51.130918 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:51.136051 (+ 5133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:51.141895 (+ 5844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:51.141902 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:51.141906 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:51.142095 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:51.142626 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:51.142633 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:51.142639 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:51.143025 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:51.143284 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:51.143870 (+ 586us) server_negotiation.cc:299] Negotiation successful
0924 11:39:51.144138 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 11:39:51.146605 30060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:51.106381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39684 (local address 127.6.212.60:34435)
0924 11:39:51.106574 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:51.106580 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:51.111405 (+ 4825us) server_negotiation.cc:407] Connection header received
0924 11:39:51.112406 (+ 1001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:51.112411 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:51.112520 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:51.112600 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:51.119409 (+ 6809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:51.120534 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:51.124812 (+ 4278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:51.125468 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:51.125586 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:51.137041 (+ 11455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:51.137049 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:51.137053 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:39:51.137098 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:51.140991 (+ 3893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:51.142657 (+ 1666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:51.142666 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:51.142671 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:51.142833 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:51.143276 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:51.143281 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:51.143285 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:51.143733 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:51.143964 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:51.145937 (+ 1973us) server_negotiation.cc:299] Negotiation successful
0924 11:39:51.146188 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":57}
I20260924 11:39:51.151732 30062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:51.105115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35294 (local address 127.6.212.62:36299)
0924 11:39:51.105311 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:51.105318 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:51.106750 (+ 1432us) server_negotiation.cc:407] Connection header received
0924 11:39:51.107163 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:51.107169 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:51.107250 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:51.107330 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:51.112210 (+ 4880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:51.113348 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:51.117302 (+ 3954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:51.118275 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:51.118402 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:51.143457 (+ 25055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:51.143472 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:51.143479 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:39:51.143540 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:51.147598 (+ 4058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:51.148429 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:51.148436 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:51.148439 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:51.148635 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:51.149083 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:51.149090 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:51.149095 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:51.149639 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:51.149941 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:51.150539 (+ 598us) server_negotiation.cc:299] Negotiation successful
0924 11:39:51.150968 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Sep 24 11:39:51 dist-test-slave-cq7c krb5kdc[29829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790249990, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.212.61@KRBTEST.COM for kudu/127.6.212.60@KRBTEST.COM
W20260924 11:39:51.153956 30099 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 95e2baa7c71c40c785fcf6d604c22eee: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 11:39:51.154067 30099 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 11:39:51.154220 30100 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260924 11:39:51.154596 30011 tablet.cc:2369] T 00000000000000000000000000000000 P 43f24860daab40238daa7439e3d68c1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 11:39:51.156608 30100 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 11:39:51.161057 29985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:51.126686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.212.60:34435 (local address 127.0.0.1:39698)
0924 11:39:51.127178 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0924 11:39:51.127203 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 11:39:51.127332 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 11:39:51.141916 (+ 14584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 11:39:51.141923 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 11:39:51.142378 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0924 11:39:51.142579 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:51.142593 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:51.147041 (+ 4448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:51.147050 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:51.148546 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 11:39:51.148562 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 11:39:51.149575 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 11:39:51.149584 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 11:39:51.149818 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:51.150445 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 11:39:51.150462 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 11:39:51.152884 (+ 2422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 11:39:51.158132 (+ 5248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:39:51.158137 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:39:51.158142 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:39:51.158699 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:39:51.159194 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 11:39:51.159201 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 11:39:51.159205 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 11:39:51.159417 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 11:39:51.160040 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 11:39:51.160045 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 11:39:51.160205 (+ 160us) client_negotiation.cc:769] Sending connection context
0924 11:39:51.160338 (+ 133us) client_negotiation.cc:240] Negotiation successful
0924 11:39:51.160640 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323}
I20260924 11:39:51.162063 30102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:51.127576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39698 (local address 127.6.212.60:34435)
0924 11:39:51.141441 (+ 13865us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:51.141447 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:51.141477 (+ 30us) server_negotiation.cc:407] Connection header received
0924 11:39:51.141532 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:51.141539 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:51.141624 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:51.141866 (+ 242us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:51.143420 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:51.146622 (+ 3202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:51.148737 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:51.149387 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:51.150137 (+ 750us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:51.153188 (+ 3051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 11:39:51.153195 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 11:39:51.153199 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 11:39:51.153235 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 11:39:51.157958 (+ 4723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:51.158862 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:51.158867 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:51.158873 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:51.159030 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 11:39:51.159542 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 11:39:51.159547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 11:39:51.159550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 11:39:51.159884 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 11:39:51.161307 (+ 1423us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:51.161434 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 11:39:51.161655 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":13685,"thread_start_us":14105,"threads_started":1}
I20260924 11:39:51.130985 29842 master_runner.cc:186] Error getting master registration for 127.6.212.61:44305: OK, instance_id { permanent_uuid: "b4561d9020204a5e9832843c71a4e7ca" instance_seqno: 1790249990392109 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 11:39:51.164279 29842 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 11:39:51.168068 29983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4561d9020204a5e9832843c71a4e7ca: Bootstrap starting.
I20260924 11:39:51.172395 29983 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4561d9020204a5e9832843c71a4e7ca: Neither blocks nor log segments found. Creating new log.
I20260924 11:39:51.173885 29983 log.cc:824] T 00000000000000000000000000000000 P b4561d9020204a5e9832843c71a4e7ca: Log is configured to *not* fsync() on all Append() calls
I20260924 11:39:51.184566 30032 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 43f24860daab40238daa7439e3d68c1f [term 1 FOLLOWER]: Refusing update from remote peer 95e2baa7c71c40c785fcf6d604c22eee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:39:51.185685 30092 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 95e2baa7c71c40c785fcf6d604c22eee [LEADER]: Connected to new peer: Peer: permanent_uuid: "43f24860daab40238daa7439e3d68c1f" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 34435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:39:51.196517 29983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4561d9020204a5e9832843c71a4e7ca: No bootstrap required, opened a new log
I20260924 11:39:51.208280 29983 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4561d9020204a5e9832843c71a4e7ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e2baa7c71c40c785fcf6d604c22eee" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36299 } } peers { permanent_uuid: "b4561d9020204a5e9832843c71a4e7ca" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 44305 } } peers { permanent_uuid: "43f24860daab40238daa7439e3d68c1f" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 34435 } }
I20260924 11:39:51.208746 29983 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4561d9020204a5e9832843c71a4e7ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4561d9020204a5e9832843c71a4e7ca, State: Initialized, Role: FOLLOWER
I20260924 11:39:51.209324 29983 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4561d9020204a5e9832843c71a4e7ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e2baa7c71c40c785fcf6d604c22eee" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36299 } } peers { permanent_uuid: "b4561d9020204a5e9832843c71a4e7ca" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 44305 } } peers { permanent_uuid: "43f24860daab40238daa7439e3d68c1f" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 34435 } }
I20260924 11:39:51.212329 30107 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4561d9020204a5e9832843c71a4e7ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e2baa7c71c40c785fcf6d604c22eee" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36299 } } peers { permanent_uuid: "b4561d9020204a5e9832843c71a4e7ca" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 44305 } } peers { permanent_uuid: "43f24860daab40238daa7439e3d68c1f" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 34435 } } }
I20260924 11:39:51.213330 30107 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4561d9020204a5e9832843c71a4e7ca [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:39:51.213474 30065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43f24860daab40238daa7439e3d68c1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95e2baa7c71c40c785fcf6d604c22eee. Latest consensus state: current_term: 1 leader_uuid: "95e2baa7c71c40c785fcf6d604c22eee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e2baa7c71c40c785fcf6d604c22eee" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36299 } } peers { permanent_uuid: "b4561d9020204a5e9832843c71a4e7ca" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 44305 } } peers { permanent_uuid: "43f24860daab40238daa7439e3d68c1f" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 34435 } } }
I20260924 11:39:51.213773 30065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43f24860daab40238daa7439e3d68c1f [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:39:51.213579 29983 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4561d9020204a5e9832843c71a4e7ca [sys.catalog]: configured and running, proceeding with master startup.
I20260924 11:39:51.213634 29958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "95e2baa7c71c40c785fcf6d604c22eee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b4561d9020204a5e9832843c71a4e7ca"
I20260924 11:39:51.214318 29957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4561d9020204a5e9832843c71a4e7ca [term 0 FOLLOWER]: Advancing to term 1
I20260924 11:39:51.218852 29957 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b4561d9020204a5e9832843c71a4e7ca [term 1 FOLLOWER]: Refusing update from remote peer 95e2baa7c71c40c785fcf6d604c22eee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 11:39:51.221350 30103 mvcc.cc:205] Tried to move back new op lower bound from 7332863963880615936 to 7332863963575717888. Current Snapshot: MvccSnapshot[applied={T|T < 7332863963880615936}]
I20260924 11:39:51.223232 30105 mvcc.cc:205] Tried to move back new op lower bound from 7332863963880615936 to 7332863963575717888. Current Snapshot: MvccSnapshot[applied={T|T < 7332863963880615936}]
I20260924 11:39:51.226328 29958 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b4561d9020204a5e9832843c71a4e7ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95e2baa7c71c40c785fcf6d604c22eee in term 1.
I20260924 11:39:51.228603 30079 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 95e2baa7c71c40c785fcf6d604c22eee [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4561d9020204a5e9832843c71a4e7ca" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 11:39:51.230576 30092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95e2baa7c71c40c785fcf6d604c22eee [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "95e2baa7c71c40c785fcf6d604c22eee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e2baa7c71c40c785fcf6d604c22eee" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36299 } } peers { permanent_uuid: "b4561d9020204a5e9832843c71a4e7ca" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 44305 } } peers { permanent_uuid: "43f24860daab40238daa7439e3d68c1f" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 34435 } } }
I20260924 11:39:51.230865 30092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95e2baa7c71c40c785fcf6d604c22eee [sys.catalog]: This master's current role is: LEADER
I20260924 11:39:51.233525 30100 catalog_manager.cc:1347] Generated new cluster ID: 8284b81ee23c45b1bf6465c6fcd09ac0
I20260924 11:39:51.233642 30100 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 11:39:51.239691 30065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43f24860daab40238daa7439e3d68c1f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "95e2baa7c71c40c785fcf6d604c22eee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e2baa7c71c40c785fcf6d604c22eee" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36299 } } peers { permanent_uuid: "b4561d9020204a5e9832843c71a4e7ca" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 44305 } } peers { permanent_uuid: "43f24860daab40238daa7439e3d68c1f" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 34435 } } }
I20260924 11:39:51.240041 30065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43f24860daab40238daa7439e3d68c1f [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:39:51.266924 30107 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4561d9020204a5e9832843c71a4e7ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95e2baa7c71c40c785fcf6d604c22eee. Latest consensus state: current_term: 1 leader_uuid: "95e2baa7c71c40c785fcf6d604c22eee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e2baa7c71c40c785fcf6d604c22eee" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36299 } } peers { permanent_uuid: "b4561d9020204a5e9832843c71a4e7ca" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 44305 } } peers { permanent_uuid: "43f24860daab40238daa7439e3d68c1f" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 34435 } } }
I20260924 11:39:51.267244 30107 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4561d9020204a5e9832843c71a4e7ca [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:39:51.267630 30115 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4561d9020204a5e9832843c71a4e7ca [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "95e2baa7c71c40c785fcf6d604c22eee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e2baa7c71c40c785fcf6d604c22eee" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36299 } } peers { permanent_uuid: "b4561d9020204a5e9832843c71a4e7ca" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 44305 } } peers { permanent_uuid: "43f24860daab40238daa7439e3d68c1f" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 34435 } } }
I20260924 11:39:51.268054 30115 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4561d9020204a5e9832843c71a4e7ca [sys.catalog]: This master's current role is: FOLLOWER
I20260924 11:39:51.276652 30092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95e2baa7c71c40c785fcf6d604c22eee [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "95e2baa7c71c40c785fcf6d604c22eee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95e2baa7c71c40c785fcf6d604c22eee" member_type: VOTER last_known_addr { host: "127.6.212.62" port: 36299 } } peers { permanent_uuid: "b4561d9020204a5e9832843c71a4e7ca" member_type: VOTER last_known_addr { host: "127.6.212.61" port: 44305 } } peers { permanent_uuid: "43f24860daab40238daa7439e3d68c1f" member_type: VOTER last_known_addr { host: "127.6.212.60" port: 34435 } } }
I20260924 11:39:51.277056 30092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95e2baa7c71c40c785fcf6d604c22eee [sys.catalog]: This master's current role is: LEADER
I20260924 11:39:51.288561 30100 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 11:39:51.291177 30100 catalog_manager.cc:1504] Loading token signing keys...
I20260924 11:39:51.328370 30100 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95e2baa7c71c40c785fcf6d604c22eee: Generated new TSK 0
I20260924 11:39:51.329823 30100 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 11:39:51.367839 30126 catalog_manager.cc:1259] Loaded cluster ID: 8284b81ee23c45b1bf6465c6fcd09ac0
I20260924 11:39:51.367966 30126 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b4561d9020204a5e9832843c71a4e7ca: loading cluster ID for follower catalog manager: success
I20260924 11:39:51.376597 30126 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b4561d9020204a5e9832843c71a4e7ca: acquiring CA information for follower catalog manager: success
I20260924 11:39:51.379280 30126 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b4561d9020204a5e9832843c71a4e7ca: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 11:39:51.498473 30062 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:39:51.482210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35306 (local address 127.6.212.62:36299)
0924 11:39:51.482607 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:51.482613 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:51.496042 (+ 13429us) server_negotiation.cc:407] Connection header received
0924 11:39:51.496413 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:51.496422 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:51.496546 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:51.496642 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:51.498164 (+ 1522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35306: BlockingRecv error: recv got EOF from 127.0.0.1:35306 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260924 11:39:51.501040 29987 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:39:51.483625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52104 (local address 127.6.212.61:44305)
0924 11:39:51.483819 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:51.483825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:51.498219 (+ 14394us) server_negotiation.cc:407] Connection header received
0924 11:39:51.499618 (+ 1399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:51.499623 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:51.499701 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:51.499780 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:51.500790 (+ 1010us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52104: BlockingRecv error: recv got EOF from 127.0.0.1:52104 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260924 11:39:51.501031 30102 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:39:51.499151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39710 (local address 127.6.212.60:34435)
0924 11:39:51.499325 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:51.499332 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:51.499354 (+ 22us) server_negotiation.cc:407] Connection header received
0924 11:39:51.499586 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:51.499592 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:51.499673 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:51.499746 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:51.500790 (+ 1044us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39710: BlockingRecv error: recv got EOF from 127.0.0.1:39710 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 11:39:51.552788 29987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:51.533359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52112 (local address 127.6.212.61:44305)
0924 11:39:51.533578 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:51.533590 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:51.543941 (+ 10351us) server_negotiation.cc:407] Connection header received
0924 11:39:51.544622 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:51.544629 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:51.544910 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:51.545070 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:39:51.545839 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:51.547004 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:51.549052 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:51.549733 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:51.549833 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:51.550941 (+ 1108us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:39:51.551885 (+ 944us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:39:51.552012 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:51.552267 (+ 255us) server_negotiation.cc:299] Negotiation successful
0924 11:39:51.552402 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260924 11:39:51.610069 30102 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:39:51.601466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39718 (local address 127.6.212.60:34435)
0924 11:39:51.601662 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:51.601670 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:51.608031 (+ 6361us) server_negotiation.cc:407] Connection header received
0924 11:39:51.608316 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:51.608323 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:51.608478 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:51.608587 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:51.609767 (+ 1180us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39718: BlockingRecv error: recv got EOF from 127.0.0.1:39718 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 11:39:51.663527 29987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:51.645324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52126 (local address 127.6.212.61:44305)
0924 11:39:51.645619 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:51.645630 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:51.656134 (+ 10504us) server_negotiation.cc:407] Connection header received
0924 11:39:51.656523 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:51.656531 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:51.656631 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:51.656705 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:39:51.657524 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:51.658738 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:51.660919 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:51.661704 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:51.661809 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:51.662520 (+ 711us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:39:51.662691 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:39:51.662756 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:51.663002 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 11:39:51.663100 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
W20260924 11:39:51.717228 30102 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:39:51.697446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39728 (local address 127.6.212.60:34435)
0924 11:39:51.697677 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:51.697684 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:51.715130 (+ 17446us) server_negotiation.cc:407] Connection header received
0924 11:39:51.715745 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:51.715753 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:51.715853 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:51.715929 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:51.716993 (+ 1064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39728: BlockingRecv error: recv got EOF from 127.0.0.1:39728 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 11:39:51.777539 29987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:51.763544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52128 (local address 127.6.212.61:44305)
0924 11:39:51.763773 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:51.763783 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:51.772476 (+ 8693us) server_negotiation.cc:407] Connection header received
0924 11:39:51.772733 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:51.772743 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:51.772890 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:51.773002 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:39:51.773467 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:51.774214 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:51.775696 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:51.776163 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:51.776299 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:51.776730 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:39:51.776854 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:39:51.776955 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:51.777093 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 11:39:51.777180 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260924 11:39:51.785943 30102 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:39:51.783400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39740 (local address 127.6.212.60:34435)
0924 11:39:51.783648 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:51.783658 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:51.784308 (+ 650us) server_negotiation.cc:407] Connection header received
0924 11:39:51.784603 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:51.784609 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:51.784704 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:51.784778 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:51.785757 (+ 979us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39740: BlockingRecv error: recv got EOF from 127.0.0.1:39740 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 11:39:51.799104 29987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:51.794003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52130 (local address 127.6.212.61:44305)
0924 11:39:51.794274 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:51.794280 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:51.794300 (+ 20us) server_negotiation.cc:407] Connection header received
0924 11:39:51.794517 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:51.794522 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:51.794589 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:51.794641 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:39:51.795134 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:51.796003 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:51.797428 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:51.797886 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:51.798024 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:51.798392 (+ 368us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:39:51.798500 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:39:51.798600 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:51.798737 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 11:39:51.798814 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
W20260924 11:39:51.811834 30102 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:39:51.809431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39742 (local address 127.6.212.60:34435)
0924 11:39:51.809720 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:51.809729 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:51.810286 (+ 557us) server_negotiation.cc:407] Connection header received
0924 11:39:51.810488 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:51.810494 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:51.810583 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:51.810691 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:51.811595 (+ 904us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39742: BlockingRecv error: recv got EOF from 127.0.0.1:39742 (error 108)
Metrics: {"server-negotiator.queue_time_us":132}
I20260924 11:39:51.829579 29987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:51.823292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52136 (local address 127.6.212.61:44305)
0924 11:39:51.823544 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:51.823554 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:51.823921 (+ 367us) server_negotiation.cc:407] Connection header received
0924 11:39:51.824178 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:51.824185 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:51.824286 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:51.824397 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:39:51.824796 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:51.825999 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:51.827698 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:51.828128 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:51.828292 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:51.828739 (+ 447us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:39:51.828865 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:39:51.828980 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:51.829144 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 11:39:51.829245 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260924 11:39:51.838218 30102 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:39:51.836160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39750 (local address 127.6.212.60:34435)
0924 11:39:51.836351 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:51.836358 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:51.836726 (+ 368us) server_negotiation.cc:407] Connection header received
0924 11:39:51.836893 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:51.836900 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:51.837017 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:51.837125 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:51.837995 (+ 870us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39750: BlockingRecv error: recv got EOF from 127.0.0.1:39750 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 11:39:51.864413 29987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:51.857956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52140 (local address 127.6.212.61:44305)
0924 11:39:51.858202 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:51.858211 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:51.858532 (+ 321us) server_negotiation.cc:407] Connection header received
0924 11:39:51.858716 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:51.858722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:51.858819 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:51.858950 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:39:51.859412 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:51.860668 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:51.862154 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:51.862827 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:51.862975 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:51.863497 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:39:51.863660 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:39:51.863764 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:51.864004 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 11:39:51.864102 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260924 11:39:51.873442 30102 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:39:51.871390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39752 (local address 127.6.212.60:34435)
0924 11:39:51.871616 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:51.871622 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:51.871929 (+ 307us) server_negotiation.cc:407] Connection header received
0924 11:39:51.872153 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:51.872160 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:51.872259 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:51.872379 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:51.873188 (+ 809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39752: BlockingRecv error: recv got EOF from 127.0.0.1:39752 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
I20260924 11:39:51.919768 29987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:51.912723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52144 (local address 127.6.212.61:44305)
0924 11:39:51.912968 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:51.912974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:51.913491 (+ 517us) server_negotiation.cc:407] Connection header received
0924 11:39:51.913725 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:51.913731 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:51.913820 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:51.913919 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:39:51.914374 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:51.915559 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:51.917577 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:51.918262 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:51.918385 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:51.918883 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:39:51.919018 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:39:51.919092 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:51.919314 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 11:39:51.919443 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260924 11:39:51.933081 30102 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:39:51.929450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39768 (local address 127.6.212.60:34435)
0924 11:39:51.929615 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:51.929621 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:51.931562 (+ 1941us) server_negotiation.cc:407] Connection header received
0924 11:39:51.931795 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:51.931800 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:51.931909 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:51.932039 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:51.932814 (+ 775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39768: BlockingRecv error: recv got EOF from 127.0.0.1:39768 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 11:39:52.006047 29987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:52.000556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52160 (local address 127.6.212.61:44305)
0924 11:39:52.000820 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:52.000829 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:52.001147 (+ 318us) server_negotiation.cc:407] Connection header received
0924 11:39:52.001405 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:52.001411 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:52.001510 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:52.001617 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:39:52.002046 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:52.002855 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:52.004282 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:52.004720 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:52.004884 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:52.005286 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:39:52.005410 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:39:52.005514 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:52.005661 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 11:39:52.005744 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20260924 11:39:52.014017 30102 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 11:39:52.011959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39770 (local address 127.6.212.60:34435)
0924 11:39:52.012217 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:52.012226 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:52.012566 (+ 340us) server_negotiation.cc:407] Connection header received
0924 11:39:52.012790 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:52.012795 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:52.012871 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:52.012982 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 11:39:52.013795 (+ 813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39770: BlockingRecv error: recv got EOF from 127.0.0.1:39770 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 11:39:52.022209 30076 catalog_manager.cc:1259] Loaded cluster ID: 8284b81ee23c45b1bf6465c6fcd09ac0
I20260924 11:39:52.022312 30076 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 43f24860daab40238daa7439e3d68c1f: loading cluster ID for follower catalog manager: success
I20260924 11:39:52.024950 30076 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 43f24860daab40238daa7439e3d68c1f: acquiring CA information for follower catalog manager: success
I20260924 11:39:52.026715 30076 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 43f24860daab40238daa7439e3d68c1f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 11:39:52.152099 29987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:52.146233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52176 (local address 127.6.212.61:44305)
0924 11:39:52.146510 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:52.146520 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:52.146873 (+ 353us) server_negotiation.cc:407] Connection header received
0924 11:39:52.147148 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:52.147153 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:52.147231 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:52.147470 (+ 239us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:39:52.147945 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:52.148702 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:52.150191 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:52.150640 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:52.150791 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:52.151264 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:39:52.151453 (+ 189us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:39:52.151562 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:52.151699 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 11:39:52.151788 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260924 11:39:52.165411 30102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:52.158916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39772 (local address 127.6.212.60:34435)
0924 11:39:52.159175 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:52.159182 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:52.159547 (+ 365us) server_negotiation.cc:407] Connection header received
0924 11:39:52.159647 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:52.159651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:52.159818 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:52.159962 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:39:52.160364 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:52.161174 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:52.162622 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:52.163213 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:52.163458 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:52.163992 (+ 534us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:39:52.164755 (+ 763us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:39:52.164874 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:52.165021 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 11:39:52.165089 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 11:39:52.182878 29987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:52.172802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52184 (local address 127.6.212.61:44305)
0924 11:39:52.173112 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:52.173121 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:52.176439 (+ 3318us) server_negotiation.cc:407] Connection header received
0924 11:39:52.176783 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:52.176793 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:52.176925 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:52.176986 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:39:52.177813 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:52.178877 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:52.180738 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:52.181364 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:52.181490 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:52.181995 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:39:52.182145 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:39:52.182239 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:52.182436 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 11:39:52.182562 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 11:39:52.182993 30102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:52.173754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39788 (local address 127.6.212.60:34435)
0924 11:39:52.173921 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:52.173925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:52.176656 (+ 2731us) server_negotiation.cc:407] Connection header received
0924 11:39:52.176954 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:52.176961 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:52.177059 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:52.177152 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:39:52.177711 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:52.178452 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:52.180608 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:52.181116 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:52.181265 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:52.182029 (+ 764us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:39:52.182182 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:39:52.182267 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:52.182503 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 11:39:52.182613 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 11:39:52.184943 30230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 11:39:52.172504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35322 (local address 127.6.212.62:36299)
0924 11:39:52.175937 (+ 3433us) server_negotiation.cc:206] Beginning negotiation
0924 11:39:52.175947 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 11:39:52.175987 (+ 40us) server_negotiation.cc:407] Connection header received
0924 11:39:52.176160 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 11:39:52.176170 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 11:39:52.176445 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 11:39:52.176651 (+ 206us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 11:39:52.177714 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:52.178867 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:52.181232 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 11:39:52.181886 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 11:39:52.182020 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 11:39:52.182828 (+ 808us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 11:39:52.183321 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 11:39:52.183456 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 11:39:52.184609 (+ 1153us) server_negotiation.cc:299] Negotiation successful
0924 11:39:52.184702 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3268,"thread_start_us":3093,"threads_started":1}
I20260924 11:39:52.189651 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 29842
I20260924 11:39:52.190707 29904 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:39:52.301424 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 29842
W20260924 11:39:52.323316 30009 connection.cc:535] server connection from 127.0.0.1:39674 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 11:39:52.324014 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 29916
I20260924 11:39:52.325194 29978 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:39:52.436169 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 29916
I20260924 11:39:52.457182 6992 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task75Snja/build/asan/bin/kudu pid 29990
I20260924 11:39:52.458477 30052 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 11:39:52.565940 6992 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task75Snja/build/asan/bin/kudu with pid 29990
2026-09-24T11:39:52Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4548 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9082 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (375468 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST